[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:22 dist-test-slave-srst krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:22 dist-test-slave-srst krb5kdc[395](info): set up 2 sockets
Sep 29 11:20:22 dist-test-slave-srst krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 11:20:23.713483 388 mini_kdc.cc:121] Time spent starting KDC: real 1.764s user 0.013s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:23 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:23.912087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35491
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:24.228684 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:24.229017 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:24.229183 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:24.237576 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:24.237710 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:24.237787 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:24.237864 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:24.237938 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:24.247978 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:24.249408 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:24.250723 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:24.262815 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:24.262791 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:24.262738 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:24.263424 408 server_base.cc:1034] running on GCE node
I20260929 11:20:24.264580 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:24.266067 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:24.267308 408 hybrid_clock.cc:648] HybridClock initialized: now 1790680824267224 us; error 92 us; skew 500 ppm
I20260929 11:20:24.267645 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:24 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:24.270728 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:24.272070 408 webserver.cc:460] Webserver started at http://127.0.97.62:44669/ using document root <none> and password file <none>
I20260929 11:20:24.272748 408 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:24.272847 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:24.273087 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:24.275837 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "975abcbdc83045448b35a5abeaec2380"
format_stamp: "Formatted at 2026-09-29 11:20:24 on dist-test-slave-srst"
server_key: "19558cbcd26d70b39c42584fb48cc1c8"
server_key_iv: "5f2c7a251198cbe9f6c1bbc175678438"
server_key_version: "encryptionkey@0"
I20260929 11:20:24.276536 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "975abcbdc83045448b35a5abeaec2380"
format_stamp: "Formatted at 2026-09-29 11:20:24 on dist-test-slave-srst"
server_key: "19558cbcd26d70b39c42584fb48cc1c8"
server_key_iv: "5f2c7a251198cbe9f6c1bbc175678438"
server_key_version: "encryptionkey@0"
I20260929 11:20:24.281489 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:20:24.284786 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:24.286136 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:20:24.286324 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "975abcbdc83045448b35a5abeaec2380"
format_stamp: "Formatted at 2026-09-29 11:20:24 on dist-test-slave-srst"
server_key: "19558cbcd26d70b39c42584fb48cc1c8"
server_key_iv: "5f2c7a251198cbe9f6c1bbc175678438"
server_key_version: "encryptionkey@0"
I20260929 11:20:24.286538 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:24.308487 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:24.312139 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:24.312419 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:24.332086 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35491
I20260929 11:20:24.332118 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35491 every 8 connection(s)
I20260929 11:20:24.333559 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:24.337070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 408
I20260929 11:20:24.337400 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:24.337841 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:24.340022 388 external_mini_cluster.cc:1428] Setting key 337fa696f8475a99b66872659ea6ebe2
I20260929 11:20:24.353235 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380: Bootstrap starting.
I20260929 11:20:24.357510 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:24.359282 475 log.cc:824] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:24.362767 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380: No bootstrap required, opened a new log
I20260929 11:20:24.373555 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975abcbdc83045448b35a5abeaec2380" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35491 } }
I20260929 11:20:24.374032 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:24.374154 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975abcbdc83045448b35a5abeaec2380, State: Initialized, Role: FOLLOWER
I20260929 11:20:24.374895 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975abcbdc83045448b35a5abeaec2380" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35491 } }
I20260929 11:20:24.375206 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:24.375357 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:24.375564 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 11:20:24 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:24.378691 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975abcbdc83045448b35a5abeaec2380" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35491 } }
I20260929 11:20:24.379215 475 leader_election.cc:302] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 975abcbdc83045448b35a5abeaec2380; no voters:
I20260929 11:20:24.379904 475 leader_election.cc:288] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:24.380159 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:24.381597 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [term 1 LEADER]: Becoming Leader. State: Replica: 975abcbdc83045448b35a5abeaec2380, State: Running, Role: LEADER
I20260929 11:20:24.382216 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975abcbdc83045448b35a5abeaec2380" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35491 } }
I20260929 11:20:24.382551 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:24.385450 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "975abcbdc83045448b35a5abeaec2380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975abcbdc83045448b35a5abeaec2380" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35491 } } }
I20260929 11:20:24.385793 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:24.390733 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 975abcbdc83045448b35a5abeaec2380. Latest consensus state: current_term: 1 leader_uuid: "975abcbdc83045448b35a5abeaec2380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975abcbdc83045448b35a5abeaec2380" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35491 } } }
I20260929 11:20:24.391454 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:24.392341 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:24.344801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.97.62:35491)
0929 11:20:24.345725 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:24.345742 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:24.347536 (+ 1794us) server_negotiation.cc:407] Connection header received
0929 11:20:24.360840 (+ 13304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:24.360862 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:24.361235 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:24.361561 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:24.365734 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:24.367107 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:24.369676 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:24.370736 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:24.370876 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:24.378439 (+ 7563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:24.378447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:24.378461 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:24.378507 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:24.382813 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:24.383949 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:24.383964 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:24.383977 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:24.384250 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:24.385078 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:24.385087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:24.385093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:24.386090 (+ 997us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:24.386481 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:24.389946 (+ 3465us) server_negotiation.cc:299] Negotiation successful
0929 11:20:24.390490 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":366,"threads_started":1}
I20260929 11:20:24.399492 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:24.403631 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:24.411262 485 catalog_manager.cc:1347] Generated new cluster ID: 434614eecf1441da9dc309f2f4752611
I20260929 11:20:24.411368 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:24.431679 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:24.433000 485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:24.442584 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 975abcbdc83045448b35a5abeaec2380: Generated new TSK 0
I20260929 11:20:24.443544 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:24.493151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35491
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:24.813931 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:24.814234 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:24.814397 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:24.821864 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:24.821964 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:24.822181 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:24.831600 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35491
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:24.832904 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:24.834227 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:24.847414 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:24.847456 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:24.847373 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:24.848197 501 server_base.cc:1034] running on GCE node
I20260929 11:20:24.848868 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:24.850284 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:24.851518 501 hybrid_clock.cc:648] HybridClock initialized: now 1790680824851484 us; error 49 us; skew 500 ppm
I20260929 11:20:24.851825 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:24 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:24.854812 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:24.856139 501 webserver.cc:460] Webserver started at http://127.0.97.1:45021/ using document root <none> and password file <none>
I20260929 11:20:24.856870 501 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:24.857007 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:24.857322 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:24.860069 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "b049840a46ff410eb09f2dd649f0786a"
format_stamp: "Formatted at 2026-09-29 11:20:24 on dist-test-slave-srst"
server_key: "ed38abc086e7a592f2e201c5bc60010b"
server_key_iv: "145cae5771524f5e50020b5a3aa0ad3d"
server_key_version: "encryptionkey@0"
I20260929 11:20:24.860882 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b049840a46ff410eb09f2dd649f0786a"
format_stamp: "Formatted at 2026-09-29 11:20:24 on dist-test-slave-srst"
server_key: "ed38abc086e7a592f2e201c5bc60010b"
server_key_iv: "145cae5771524f5e50020b5a3aa0ad3d"
server_key_version: "encryptionkey@0"
I20260929 11:20:24.865880 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:20:24.869192 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:24.870455 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:20:24.870635 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "b049840a46ff410eb09f2dd649f0786a"
format_stamp: "Formatted at 2026-09-29 11:20:24 on dist-test-slave-srst"
server_key: "ed38abc086e7a592f2e201c5bc60010b"
server_key_iv: "145cae5771524f5e50020b5a3aa0ad3d"
server_key_version: "encryptionkey@0"
I20260929 11:20:24.870827 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:24.897758 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:24.901943 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:24.902213 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:24.903077 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:24.904289 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:24.904382 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:24.904538 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:24.904645 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:24.939085 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35445
I20260929 11:20:24.939136 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35445 every 8 connection(s)
I20260929 11:20:24.940303 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:24.949041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 501
I20260929 11:20:24.949246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:24.949882 388 external_mini_cluster.cc:1428] Setting key c71281eaaccd8fb8d8c82bef964a2b21
Sep 29 11:20:24 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:24.962953 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:24.943075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36133 (local address 127.0.97.62:35491)
0929 11:20:24.943516 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:24.943524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:24.944981 (+ 1457us) server_negotiation.cc:407] Connection header received
0929 11:20:24.946783 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:24.946787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:24.946863 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:24.946970 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:24.948310 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:24.949337 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:24.950926 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:24.951332 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:24.951478 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:24.955956 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:24.955962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:24.955965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:24.956019 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:24.959424 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:24.960261 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:24.960269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:24.960276 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:24.960431 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:24.960905 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:24.960913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:24.960919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:24.961347 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:24.961501 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:24.962217 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 11:20:24.962470 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260929 11:20:24.963558 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:24.943650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35491 (local address 127.0.97.1:36133)
0929 11:20:24.944818 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:24.944862 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:24.946477 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:24.947191 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:24.947199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:24.947650 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:24.948165 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:24.948181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:24.949476 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:24.949479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:24.950795 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:24.950806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:24.951685 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:24.951692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:24.951962 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:24.952706 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:24.952727 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:24.955755 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:24.959581 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:24.959593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:24.959612 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:24.960138 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:24.960542 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:24.960545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:24.960548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:24.960794 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:24.961552 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:24.961558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:24.961846 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 11:20:24.962095 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 11:20:24.962456 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":668,"threads_started":1}
I20260929 11:20:24.965214 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35491
I20260929 11:20:24.965505 629 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:24.966180 629 heartbeater.cc:511] Master 127.0.97.62:35491 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:20:24.968194 440 ts_manager.cc:194] Registered new tserver with Master: b049840a46ff410eb09f2dd649f0786a (127.0.97.1:35445)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 11:20:24.969760 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:24.991740 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35491
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:25.317307 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:25.317631 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:25.317800 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:25.325284 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:25.325419 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:25.325598 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:25.335177 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35491
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:25.336393 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:25.337726 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:25.353011 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:25.354079 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:25.354100 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:25.354492 635 server_base.cc:1034] running on GCE node
I20260929 11:20:25.355141 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:25.356215 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:25.357439 635 hybrid_clock.cc:648] HybridClock initialized: now 1790680825357390 us; error 40 us; skew 500 ppm
I20260929 11:20:25.357782 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:25 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:25.361407 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:25.362802 635 webserver.cc:460] Webserver started at http://127.0.97.2:34107/ using document root <none> and password file <none>
I20260929 11:20:25.363471 635 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:25.363615 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:25.363917 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:25.366758 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "761a0c7584b849c3835eca1235651df2"
format_stamp: "Formatted at 2026-09-29 11:20:25 on dist-test-slave-srst"
server_key: "94e78062b8e2ae0bd3324e0f004e8a0e"
server_key_iv: "dd2159c700d37be1d7c935bff1f1f610"
server_key_version: "encryptionkey@0"
I20260929 11:20:25.367501 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "761a0c7584b849c3835eca1235651df2"
format_stamp: "Formatted at 2026-09-29 11:20:25 on dist-test-slave-srst"
server_key: "94e78062b8e2ae0bd3324e0f004e8a0e"
server_key_iv: "dd2159c700d37be1d7c935bff1f1f610"
server_key_version: "encryptionkey@0"
I20260929 11:20:25.372382 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260929 11:20:25.375927 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:25.377203 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:25.377389 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "761a0c7584b849c3835eca1235651df2"
format_stamp: "Formatted at 2026-09-29 11:20:25 on dist-test-slave-srst"
server_key: "94e78062b8e2ae0bd3324e0f004e8a0e"
server_key_iv: "dd2159c700d37be1d7c935bff1f1f610"
server_key_version: "encryptionkey@0"
I20260929 11:20:25.377571 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:25.406167 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:25.409947 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:25.410254 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:25.411202 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:25.412461 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:25.412566 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:25.412786 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:25.412851 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:25.446919 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42291
I20260929 11:20:25.447014 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42291 every 8 connection(s)
I20260929 11:20:25.448182 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:25.456311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 635
I20260929 11:20:25.456501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:25.456944 388 external_mini_cluster.cc:1428] Setting key becdaa4892c88421f91864252a64a024
Sep 29 11:20:25 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:25.467767 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:25.450647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46531 (local address 127.0.97.62:35491)
0929 11:20:25.450813 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:25.450817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:25.451900 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 11:20:25.453844 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:25.453849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:25.453950 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:25.454059 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:25.455450 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.456172 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.457900 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.458334 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.458453 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:25.461897 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:25.461902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:25.461905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:25.461934 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:25.464859 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:25.465579 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:25.465588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:25.465591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:25.465720 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:25.466147 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:25.466154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:25.466162 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:25.466397 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:25.466545 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:25.467184 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 11:20:25.467413 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:20:25.468793 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:25.451101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35491 (local address 127.0.97.2:46531)
0929 11:20:25.451752 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:25.451788 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:25.453565 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:25.454284 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:25.454292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:25.454753 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:25.455289 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:25.455306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.456317 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.456324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:25.457756 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:25.457768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.458571 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.458581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:25.458886 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:25.459598 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:25.459619 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:25.461686 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:25.465001 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:25.465009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:25.465024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:25.465450 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:25.465842 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:25.465848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:25.465852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:25.466031 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:25.466523 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:25.466530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:25.466845 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 11:20:25.467098 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 11:20:25.467628 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":244,"threads_started":1}
I20260929 11:20:25.470353 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35491
I20260929 11:20:25.470636 763 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:25.471309 763 heartbeater.cc:511] Master 127.0.97.62:35491 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:20:25.472698 440 ts_manager.cc:194] Registered new tserver with Master: 761a0c7584b849c3835eca1235651df2 (127.0.97.2:42291)
I20260929 11:20:25.473616 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:25.495538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35491
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:25.819747 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:25.820000 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:25.820093 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:25.827196 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:25.827298 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:25.827416 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:25.837337 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35491
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:25.838563 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:25.839877 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:25.854243 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:25.854277 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:25.854323 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:25.854996 768 server_base.cc:1034] running on GCE node
I20260929 11:20:25.855638 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:25.856748 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:25.857967 768 hybrid_clock.cc:648] HybridClock initialized: now 1790680825857912 us; error 50 us; skew 500 ppm
I20260929 11:20:25.858285 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:25 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:25.861634 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:25.863145 768 webserver.cc:460] Webserver started at http://127.0.97.3:43481/ using document root <none> and password file <none>
I20260929 11:20:25.863842 768 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:25.863991 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:25.864307 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:25.867708 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "a73d3935691e41feadc8191c62dd028b"
format_stamp: "Formatted at 2026-09-29 11:20:25 on dist-test-slave-srst"
server_key: "409a74b07a7ed1a872a496e468f91cc0"
server_key_iv: "ee24c7a0372cc5c87c4fca59953c7b93"
server_key_version: "encryptionkey@0"
I20260929 11:20:25.868484 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a73d3935691e41feadc8191c62dd028b"
format_stamp: "Formatted at 2026-09-29 11:20:25 on dist-test-slave-srst"
server_key: "409a74b07a7ed1a872a496e468f91cc0"
server_key_iv: "ee24c7a0372cc5c87c4fca59953c7b93"
server_key_version: "encryptionkey@0"
I20260929 11:20:25.874285 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 11:20:25.877856 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:25.879179 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 11:20:25.879349 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "a73d3935691e41feadc8191c62dd028b"
format_stamp: "Formatted at 2026-09-29 11:20:25 on dist-test-slave-srst"
server_key: "409a74b07a7ed1a872a496e468f91cc0"
server_key_iv: "ee24c7a0372cc5c87c4fca59953c7b93"
server_key_version: "encryptionkey@0"
I20260929 11:20:25.879554 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:25.895615 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:25.899838 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:25.900120 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:25.901057 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:25.902294 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:25.902400 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:25.902544 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:25.902617 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:25.938064 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46701
I20260929 11:20:25.938115 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46701 every 8 connection(s)
I20260929 11:20:25.939725 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:25.944440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 768
I20260929 11:20:25.944658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:25.945080 388 external_mini_cluster.cc:1428] Setting key 6ab05e9a5054fb82588ebcce42d336ea
Sep 29 11:20:25 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:25.959733 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:25.942333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45005 (local address 127.0.97.62:35491)
0929 11:20:25.942542 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:25.942548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:25.943805 (+ 1257us) server_negotiation.cc:407] Connection header received
0929 11:20:25.945754 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:25.945762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:25.945841 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:25.945989 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:25.947829 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.948789 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.950769 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.951143 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.951242 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:25.954553 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:25.954559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:25.954562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:25.954593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:25.957006 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:25.957664 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:25.957667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:25.957670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:25.957802 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:25.958164 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:25.958167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:25.958169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:25.958419 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:25.958522 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:25.959176 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 11:20:25.959376 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:20:25.960983 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:25.942822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35491 (local address 127.0.97.3:45005)
0929 11:20:25.943606 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:25.943653 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:25.945476 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:25.946234 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:25.946266 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:25.946901 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:25.947650 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:25.947675 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.948932 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.948936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:25.950646 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:25.950657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:25.951326 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:25.951330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:25.951516 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:25.952118 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:25.952139 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:25.954343 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:25.957130 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:25.957137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:25.957149 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:25.957560 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:25.957902 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:25.957905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:25.957907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:25.958077 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:25.958518 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:25.958523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:25.958838 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 11:20:25.959063 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 11:20:25.959564 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":337,"threads_started":1}
I20260929 11:20:25.962332 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35491
I20260929 11:20:25.962582 896 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:25.963225 896 heartbeater.cc:511] Master 127.0.97.62:35491 requested a full tablet report, sending...
I20260929 11:20:25.964493 440 ts_manager.cc:194] Registered new tserver with Master: a73d3935691e41feadc8191c62dd028b (127.0.97.3:46701)
I20260929 11:20:25.965299 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45005
I20260929 11:20:25.973235 629 heartbeater.cc:503] Master 127.0.97.62:35491 was elected leader, sending a full tablet report...
I20260929 11:20:25.973287 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:20:25 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:26.022543 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:26.003899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.0.97.1:35445)
0929 11:20:26.009046 (+ 5147us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:26.009054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:26.009079 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:20:26.009138 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:26.009144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:26.009368 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:26.009494 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:26.010848 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.011848 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.013130 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.013683 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.013792 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:26.017606 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:26.017616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:26.017630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:26.017671 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:26.020225 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.020843 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.020850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.020853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.020965 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.021273 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.021276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.021278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.021576 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:26.021802 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:26.021991 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 11:20:26.022165 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5008,"thread_start_us":4857,"threads_started":1}
W20260929 11:20:26.023851 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46524
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:26.066075 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:26.052994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.0.97.1:35445)
0929 11:20:26.053156 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:26.053162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:26.053571 (+ 409us) server_negotiation.cc:407] Connection header received
0929 11:20:26.053692 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:26.053696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:26.053804 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:26.053910 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:26.055148 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.055842 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.057066 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.057477 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.057585 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:26.061312 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:26.061317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:26.061321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:26.061352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:26.063934 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.064487 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.064491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.064493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.064686 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.064981 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.064984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.064986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.065242 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:26.065419 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:26.065658 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 11:20:26.065826 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:20:26.067638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 501
I20260929 11:20:26.080036 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:26.068494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.0.97.1:35445)
0929 11:20:26.068723 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:26.068731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:26.069037 (+ 306us) server_negotiation.cc:407] Connection header received
0929 11:20:26.069147 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:26.069150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:26.069204 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:26.069277 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:26.070293 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.070947 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.072091 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.072543 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.072766 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:26.074993 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:26.074998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:26.075001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:26.075028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:26.077823 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.078358 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.078362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.078364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.078476 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.078783 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.078787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.078789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.079034 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:26.079221 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:26.079496 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 11:20:26.079704 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:20:26.080797 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:26.163079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 501
I20260929 11:20:26.179607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 635
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:20:26.195061 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:26.180493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.0.97.2:42291)
0929 11:20:26.181155 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:26.181162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:26.181186 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:20:26.181245 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:26.181252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:26.181459 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:26.181651 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:26.182825 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.183965 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.185300 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.185832 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.185945 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:26.189784 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:26.189796 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:26.189810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:26.189841 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:26.192529 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.193275 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.193282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.193284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.193394 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.193732 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.193735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.193737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.194031 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:26.194263 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:26.194460 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 11:20:26.194632 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":384,"threads_started":1}
I20260929 11:20:26.196290 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:26.278892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 635
I20260929 11:20:26.301258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 768
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:20:26.315847 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:26.302054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.97.3:46701)
0929 11:20:26.302668 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:26.302675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:26.302705 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:20:26.302766 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:26.302772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:26.302985 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:26.303115 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:26.304270 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.305362 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.306546 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:26.307070 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:26.307168 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:26.311130 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:26.311141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:26.311154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:26.311182 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:26.313706 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.314264 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.314271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.314273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.314382 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:26.314675 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:26.314678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:26.314680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:26.314972 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:26.315182 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:26.315393 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 11:20:26.315568 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
I20260929 11:20:26.317106 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:26.398581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 768
I20260929 11:20:26.416415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 408
I20260929 11:20:26.417878 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:26.518482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 408
2026-09-29T11:20:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4621 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[917](info): set up 2 sockets
Sep 29 11:20:26 dist-test-slave-srst krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 11:20:27.580390 388 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:27 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:27.751809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44903
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:28.062544 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:28.062825 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:28.063015 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:28.070099 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:28.070204 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:28.070290 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:28.070382 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:28.070472 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:28.080020 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:28.081394 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:28.082753 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:28.094372 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:28.094322 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:28.094285 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:28.095052 930 server_base.cc:1034] running on GCE node
I20260929 11:20:28.095814 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:28.097177 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:28.098378 930 hybrid_clock.cc:648] HybridClock initialized: now 1790680828098311 us; error 51 us; skew 500 ppm
I20260929 11:20:28.098742 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:28 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:28.106575 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:28.108156 930 webserver.cc:460] Webserver started at http://127.0.97.62:33107/ using document root <none> and password file <none>
I20260929 11:20:28.108906 930 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:28.109055 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:28.109369 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:28.112200 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "fdb04da10f90424bbb74e7e4a9cdf833"
format_stamp: "Formatted at 2026-09-29 11:20:28 on dist-test-slave-srst"
server_key: "49df884fcae7f5bf3107094d48343479"
server_key_iv: "61f05308f4b32daf0286f9ef97993cd3"
server_key_version: "encryptionkey@0"
I20260929 11:20:28.112994 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "fdb04da10f90424bbb74e7e4a9cdf833"
format_stamp: "Formatted at 2026-09-29 11:20:28 on dist-test-slave-srst"
server_key: "49df884fcae7f5bf3107094d48343479"
server_key_iv: "61f05308f4b32daf0286f9ef97993cd3"
server_key_version: "encryptionkey@0"
I20260929 11:20:28.117974 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:20:28.121302 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:28.122638 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:20:28.122812 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "fdb04da10f90424bbb74e7e4a9cdf833"
format_stamp: "Formatted at 2026-09-29 11:20:28 on dist-test-slave-srst"
server_key: "49df884fcae7f5bf3107094d48343479"
server_key_iv: "61f05308f4b32daf0286f9ef97993cd3"
server_key_version: "encryptionkey@0"
I20260929 11:20:28.123015 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:28.139233 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:28.143040 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:28.143307 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:28.161047 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44903
I20260929 11:20:28.161083 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44903 every 8 connection(s)
I20260929 11:20:28.162339 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:28.165858 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:28.166842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 930
I20260929 11:20:28.167078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:28.167865 388 external_mini_cluster.cc:1428] Setting key 63f5a265e0cddf951b2d2367621e1e53
I20260929 11:20:28.179838 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833: Bootstrap starting.
Sep 29 11:20:28 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:28.183569 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:28.184914 997 log.cc:824] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:28.187870 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833: No bootstrap required, opened a new log
I20260929 11:20:28.190089 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:28.169800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.97.62:44903)
0929 11:20:28.170557 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:28.170572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:28.170609 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:20:28.172137 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:28.172157 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:28.172534 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:28.172894 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:28.174528 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:28.175716 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:28.177577 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:28.178215 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:28.178334 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:28.183035 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:28.183042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:28.183056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:28.183095 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:28.186212 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:28.186871 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:28.186878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:28.186884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:28.187047 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:28.187403 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:28.187410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:28.187413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:28.187905 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:28.188184 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:28.188733 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 11:20:28.189094 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":361,"threads_started":1}
I20260929 11:20:28.196009 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44903 } }
I20260929 11:20:28.196399 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:28.196512 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb04da10f90424bbb74e7e4a9cdf833, State: Initialized, Role: FOLLOWER
I20260929 11:20:28.197098 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44903 } }
I20260929 11:20:28.197348 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:28.197510 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:28.197680 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:28.199931 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44903 } }
I20260929 11:20:28.200431 997 leader_election.cc:302] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb04da10f90424bbb74e7e4a9cdf833; no voters:
I20260929 11:20:28.201007 997 leader_election.cc:288] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:28.201172 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:28.201887 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [term 1 LEADER]: Becoming Leader. State: Replica: fdb04da10f90424bbb74e7e4a9cdf833, State: Running, Role: LEADER
I20260929 11:20:28.202414 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44903 } }
I20260929 11:20:28.203264 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:28.205240 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb04da10f90424bbb74e7e4a9cdf833. Latest consensus state: current_term: 1 leader_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44903 } } }
I20260929 11:20:28.205691 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:28.206064 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb04da10f90424bbb74e7e4a9cdf833" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44903 } } }
I20260929 11:20:28.206403 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:28.207568 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:28.212273 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:28.221604 1009 catalog_manager.cc:1347] Generated new cluster ID: 46c67eec581c4b27937c335b77c0b241
I20260929 11:20:28.221728 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:28.259688 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:28.261023 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:28.267091 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb04da10f90424bbb74e7e4a9cdf833: Generated new TSK 0
I20260929 11:20:28.267908 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:28.320084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44903
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:28.663125 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:28.663380 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:28.663538 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:28.670480 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:28.670569 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:28.670732 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:28.679888 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44903
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:28.681170 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:28.682459 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:28.695765 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:28.695817 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:28.695817 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:28.696900 1023 server_base.cc:1034] running on GCE node
I20260929 11:20:28.697405 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:28.698221 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:28.699414 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790680828699357 us; error 50 us; skew 500 ppm
I20260929 11:20:28.699728 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:28 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:28.702697 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:28.704120 1023 webserver.cc:460] Webserver started at http://127.0.97.1:38565/ using document root <none> and password file <none>
I20260929 11:20:28.704780 1023 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:28.704902 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:28.705229 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:28.707995 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "e850892f010f46239245e5e5d90b6b17"
format_stamp: "Formatted at 2026-09-29 11:20:28 on dist-test-slave-srst"
server_key: "06ff23719fde30ac86cbab7aa5635500"
server_key_iv: "96206f10b380a75bae05a360c200c918"
server_key_version: "encryptionkey@0"
I20260929 11:20:28.708808 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e850892f010f46239245e5e5d90b6b17"
format_stamp: "Formatted at 2026-09-29 11:20:28 on dist-test-slave-srst"
server_key: "06ff23719fde30ac86cbab7aa5635500"
server_key_iv: "96206f10b380a75bae05a360c200c918"
server_key_version: "encryptionkey@0"
I20260929 11:20:28.713819 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 11:20:28.717111 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:28.718341 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:20:28.718518 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "e850892f010f46239245e5e5d90b6b17"
format_stamp: "Formatted at 2026-09-29 11:20:28 on dist-test-slave-srst"
server_key: "06ff23719fde30ac86cbab7aa5635500"
server_key_iv: "96206f10b380a75bae05a360c200c918"
server_key_version: "encryptionkey@0"
I20260929 11:20:28.718690 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:28.744705 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:28.748710 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:28.748996 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:28.749939 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:28.751543 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:28.751637 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:28.751756 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:28.751806 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:28.783697 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37617
I20260929 11:20:28.783784 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37617 every 8 connection(s)
I20260929 11:20:28.785023 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:28.788082 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1023
I20260929 11:20:28.788295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:28.788997 388 external_mini_cluster.cc:1428] Setting key 2cd5095bb5f41a86ace181508f497f2a
Sep 29 11:20:28 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 11:20:28.806352 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:28.787679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60735 (local address 127.0.97.62:44903)
0929 11:20:28.788190 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:28.788196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:28.789491 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 11:20:28.791678 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:28.791683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:28.791779 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:28.791914 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:28.793541 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:28.794281 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:28.795854 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:28.796276 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:28.796414 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:28.800099 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:28.800104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:28.800108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:28.800138 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:28.802855 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:28.803767 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:28.803771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:28.803774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:28.803884 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:28.804482 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:28.804496 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:28.804499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:28.804861 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:28.805036 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:28.805719 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 11:20:28.805945 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260929 11:20:28.807325 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:28.788361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44903 (local address 127.0.97.1:60735)
0929 11:20:28.789292 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:28.789347 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:28.791396 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:28.792159 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:28.792168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:28.792798 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:28.793369 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:28.793385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:28.794431 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:28.794439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:28.795704 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:28.795719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:28.796497 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:28.796506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:28.796771 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:28.797499 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:28.797522 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:28.799871 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:28.803031 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:28.803048 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:28.803061 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:28.803618 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:28.804004 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:28.804009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:28.804013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:28.804213 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:28.805015 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:28.805023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:28.805330 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 11:20:28.805606 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 11:20:28.806130 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":405,"threads_started":1}
I20260929 11:20:28.809010 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44903
I20260929 11:20:28.809314 1151 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:28.810039 1151 heartbeater.cc:511] Master 127.0.97.62:44903 requested a full tablet report, sending...
I20260929 11:20:28.812150 962 ts_manager.cc:194] Registered new tserver with Master: e850892f010f46239245e5e5d90b6b17 (127.0.97.1:37617)
I20260929 11:20:28.813793 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:28.828462 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44903
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:29.132797 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:29.133059 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:29.133201 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:29.140213 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:29.140336 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:29.140515 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:29.149987 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44903
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:29.151242 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:29.152534 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:29.165002 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:29.165036 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:29.165155 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:29.166070 1157 server_base.cc:1034] running on GCE node
I20260929 11:20:29.166646 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:29.167579 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:29.168777 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790680829168730 us; error 44 us; skew 500 ppm
I20260929 11:20:29.169072 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:29 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:29.172374 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:29.174026 1157 webserver.cc:460] Webserver started at http://127.0.97.2:33159/ using document root <none> and password file <none>
I20260929 11:20:29.174821 1157 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:29.174959 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:29.175309 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:29.179016 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "4fd8322d44484e6ba9b840ab90983cda"
format_stamp: "Formatted at 2026-09-29 11:20:29 on dist-test-slave-srst"
server_key: "76ca222f7ec32b10558eb6bbcdde1d6d"
server_key_iv: "aef399a995f4dd57f21aaba737ac6842"
server_key_version: "encryptionkey@0"
I20260929 11:20:29.179968 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4fd8322d44484e6ba9b840ab90983cda"
format_stamp: "Formatted at 2026-09-29 11:20:29 on dist-test-slave-srst"
server_key: "76ca222f7ec32b10558eb6bbcdde1d6d"
server_key_iv: "aef399a995f4dd57f21aaba737ac6842"
server_key_version: "encryptionkey@0"
I20260929 11:20:29.184960 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:20:29.188158 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:29.189468 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:29.189631 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "4fd8322d44484e6ba9b840ab90983cda"
format_stamp: "Formatted at 2026-09-29 11:20:29 on dist-test-slave-srst"
server_key: "76ca222f7ec32b10558eb6bbcdde1d6d"
server_key_iv: "aef399a995f4dd57f21aaba737ac6842"
server_key_version: "encryptionkey@0"
I20260929 11:20:29.189798 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:29.214989 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:29.219285 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:29.219558 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:29.220420 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:29.221772 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:29.221841 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:29.221982 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:29.222047 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:29.255486 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44303
I20260929 11:20:29.255543 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44303 every 8 connection(s)
I20260929 11:20:29.256807 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:29.262952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1157
I20260929 11:20:29.263151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:29.263571 388 external_mini_cluster.cc:1428] Setting key 5ce0080554e9013a7fa49c91e7f43747
Sep 29 11:20:29 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:29.276214 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.259223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35763 (local address 127.0.97.62:44903)
0929 11:20:29.259392 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.259397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.260560 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 11:20:29.262317 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.262321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.262384 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.262471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:29.264080 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.265014 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.266663 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.267444 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.267599 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.270813 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:29.270817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:29.270820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:29.270848 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:29.273399 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.274073 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.274079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.274083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.274213 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.274621 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.274627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.274630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.274882 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:29.275029 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.275672 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.275876 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:20:29.277209 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.259720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44903 (local address 127.0.97.2:35763)
0929 11:20:29.260376 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.260442 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.262075 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.262699 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.262707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.263313 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:29.263910 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.263928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.265202 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.265210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.266513 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.266524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.267657 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.267662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.267848 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.268492 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:29.268513 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:29.270606 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:29.273532 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:29.273551 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:29.273566 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:29.273953 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:29.274317 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:29.274323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:29.274326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:29.274503 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:29.275009 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:29.275017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:29.275317 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.275587 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.276079 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:20:29.278586 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44903
I20260929 11:20:29.278874 1285 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:29.279536 1285 heartbeater.cc:511] Master 127.0.97.62:44903 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 11:20:29.281081 962 ts_manager.cc:194] Registered new tserver with Master: 4fd8322d44484e6ba9b840ab90983cda (127.0.97.2:44303)
I20260929 11:20:29.281905 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:29.301352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44903
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:29.611203 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:29.611469 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:29.611624 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:29.618742 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:29.618834 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:29.619001 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:29.628502 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44903
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:29.629730 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:29.631042 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:29.643810 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:29.643961 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:29.644414 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:29.645126 1290 server_base.cc:1034] running on GCE node
I20260929 11:20:29.645706 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:29.646592 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:29.647817 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790680829647778 us; error 56 us; skew 500 ppm
I20260929 11:20:29.648126 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:29 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:29.651283 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:29.652709 1290 webserver.cc:460] Webserver started at http://127.0.97.3:40137/ using document root <none> and password file <none>
I20260929 11:20:29.653364 1290 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:29.653506 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:29.653810 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:29.656560 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31"
format_stamp: "Formatted at 2026-09-29 11:20:29 on dist-test-slave-srst"
server_key: "f0ffbdd925922fa8696a1bca60147f91"
server_key_iv: "97ade66af664309972bfd984ed9f07c6"
server_key_version: "encryptionkey@0"
I20260929 11:20:29.657387 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31"
format_stamp: "Formatted at 2026-09-29 11:20:29 on dist-test-slave-srst"
server_key: "f0ffbdd925922fa8696a1bca60147f91"
server_key_iv: "97ade66af664309972bfd984ed9f07c6"
server_key_version: "encryptionkey@0"
I20260929 11:20:29.662269 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:20:29.665526 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:29.666774 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:20:29.666946 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31"
format_stamp: "Formatted at 2026-09-29 11:20:29 on dist-test-slave-srst"
server_key: "f0ffbdd925922fa8696a1bca60147f91"
server_key_iv: "97ade66af664309972bfd984ed9f07c6"
server_key_version: "encryptionkey@0"
I20260929 11:20:29.667135 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:29.687111 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:29.691128 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:29.691421 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:29.692329 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:29.693665 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:29.693768 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:29.693925 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:29.693976 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:29.725943 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33487
I20260929 11:20:29.725981 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33487 every 8 connection(s)
I20260929 11:20:29.727185 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:29.735950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1290
I20260929 11:20:29.736166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:29.736652 388 external_mini_cluster.cc:1428] Setting key dad597f30fb80582434031e04a3e55bb
Sep 29 11:20:29 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:29.747051 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.729824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50461 (local address 127.0.97.62:44903)
0929 11:20:29.729993 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.729998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.731176 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 11:20:29.733460 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.733465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.733535 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.733646 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:29.735052 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.736005 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.737861 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.738261 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.738397 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.741846 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:29.741850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:29.741853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:29.741882 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:29.744416 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.745122 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.745125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.745128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.745249 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.745651 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.745654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.745656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.745888 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:29.745986 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.746648 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.746804 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:20:29.747946 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.730337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44903 (local address 127.0.97.3:50461)
0929 11:20:29.731000 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.731045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.733185 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.733833 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.733842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.734334 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:29.734857 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.734873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.736158 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.736164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.737716 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.737730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.738489 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.738498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.738744 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.739425 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:29.739448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:29.741651 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:29.744521 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:29.744527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:29.744538 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:29.744985 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:29.745350 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:29.745353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:29.745355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:29.745528 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:29.745982 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:29.745987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:29.746258 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.746499 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.746851 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":253,"threads_started":1}
I20260929 11:20:29.749317 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44903
I20260929 11:20:29.749609 1418 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:29.750293 1418 heartbeater.cc:511] Master 127.0.97.62:44903 requested a full tablet report, sending...
I20260929 11:20:29.751879 962 ts_manager.cc:194] Registered new tserver with Master: 42b9e3c9bd024f0d9ac62a5b1b4ffa31 (127.0.97.3:33487)
I20260929 11:20:29.752749 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50461
I20260929 11:20:29.763675 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:20:29.781530 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.770117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.0.97.62:44903)
0929 11:20:29.770297 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.770302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.770841 (+ 539us) server_negotiation.cc:407] Connection header received
0929 11:20:29.770982 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.770987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.771064 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.771169 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:29.772502 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.773240 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.774425 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.774782 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.774866 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.777079 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:29.777083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:29.777086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:29.777112 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:29.779396 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.779932 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.779936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.779938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.780043 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.780354 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.780357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.780359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.780686 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:29.780875 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.781132 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.781316 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:20:29.788451 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:29.795164 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:29.817380 1151 heartbeater.cc:503] Master 127.0.97.62:44903 was elected leader, sending a full tablet report...
I20260929 11:20:29.818694 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.806718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33487 (local address 127.0.0.1:46750)
0929 11:20:29.808255 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.808309 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.808464 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.810360 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.810367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.810395 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:29.810831 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.810848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.813082 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.813087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.815689 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.815705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.817620 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.817626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.817879 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.817941 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.818073 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.818268 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":24,"thread_start_us":2797,"threads_started":1}
I20260929 11:20:29.819317 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.807841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.0.97.1:37617)
0929 11:20:29.809356 (+ 1515us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.809367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.809960 (+ 593us) server_negotiation.cc:407] Connection header received
0929 11:20:29.810045 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.810051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.810236 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.810402 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:29.811303 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.812930 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.816604 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.817958 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.818100 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.818333 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.818849 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.819002 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":1207,"threads_started":1}
I20260929 11:20:29.819840 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.806903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.0.97.3:33487)
0929 11:20:29.809716 (+ 2813us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.809724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.809761 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:20:29.809859 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.809866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.810111 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.811053 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:29.811320 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.812932 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.815873 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.817373 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.819050 (+ 1677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.819281 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.819352 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.819520 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2648,"thread_start_us":2503,"threads_started":1}
I20260929 11:20:29.820788 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.806815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37617 (local address 127.0.0.1:49476)
0929 11:20:29.809718 (+ 2903us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.809735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.809849 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.810639 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.810645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.810670 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:29.811107 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.811119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.813081 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.813087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.816371 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.816384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.818470 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.818476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.818630 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.818665 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.819488 (+ 823us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.819785 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2783,"mutex_wait_us":696,"thread_start_us":1189,"threads_started":1}
I20260929 11:20:29.821964 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.810118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.0.97.2:44303)
0929 11:20:29.813527 (+ 3409us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.813535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.814136 (+ 601us) server_negotiation.cc:407] Connection header received
0929 11:20:29.814298 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.814308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.814544 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.814754 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:29.815379 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.816941 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.819434 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.820811 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.820947 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.821129 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.821510 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.821669 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3247,"thread_start_us":3112,"threads_started":1}
I20260929 11:20:29.821980 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.809881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44303 (local address 127.0.0.1:34146)
0929 11:20:29.813978 (+ 4097us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.814005 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.814123 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.814755 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.814760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.814787 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:29.815194 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.815204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.817080 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.817086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.819251 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.819264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.821233 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.821237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.821341 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.821361 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.821461 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.821622 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3929,"mutex_wait_us":3,"thread_start_us":3786,"threads_started":1}
I20260929 11:20:29.823879 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 07bc2d4d96184664b4b5f47712665a16 (DEFAULT_TABLE table=test-table [id=6f910dbf826442b2ab51bee69beab40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:29.824331 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 07bc2d4d96184664b4b5f47712665a16 (DEFAULT_TABLE table=test-table [id=6f910dbf826442b2ab51bee69beab40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:29.825470 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bc2d4d96184664b4b5f47712665a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:29.825130 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 07bc2d4d96184664b4b5f47712665a16 (DEFAULT_TABLE table=test-table [id=6f910dbf826442b2ab51bee69beab40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:29.825663 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bc2d4d96184664b4b5f47712665a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:29.826431 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bc2d4d96184664b4b5f47712665a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:29.839329 1436 tablet_bootstrap.cc:492] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: Bootstrap starting.
I20260929 11:20:29.840548 1437 tablet_bootstrap.cc:492] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: Bootstrap starting.
I20260929 11:20:29.842319 1438 tablet_bootstrap.cc:492] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: Bootstrap starting.
I20260929 11:20:29.842932 1436 tablet_bootstrap.cc:654] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:29.844671 1437 tablet_bootstrap.cc:654] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:29.844667 1436 log.cc:824] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:29.845403 1438 tablet_bootstrap.cc:654] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:29.845824 1437 log.cc:824] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:29.846596 1438 log.cc:824] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:29.847882 1436 tablet_bootstrap.cc:492] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: No bootstrap required, opened a new log
I20260929 11:20:29.848137 1436 ts_tablet_manager.cc:1397] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 11:20:29.849020 1437 tablet_bootstrap.cc:492] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: No bootstrap required, opened a new log
I20260929 11:20:29.849299 1437 ts_tablet_manager.cc:1397] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:20:29.851040 1438 tablet_bootstrap.cc:492] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: No bootstrap required, opened a new log
I20260929 11:20:29.851287 1438 ts_tablet_manager.cc:1397] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 11:20:29.855630 1436 raft_consensus.cc:348] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.856420 1436 raft_consensus.cc:374] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:29.856511 1436 raft_consensus.cc:729] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd8322d44484e6ba9b840ab90983cda, State: Initialized, Role: FOLLOWER
I20260929 11:20:29.857125 1436 consensus_queue.cc:260] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.859797 1285 heartbeater.cc:503] Master 127.0.97.62:44903 was elected leader, sending a full tablet report...
I20260929 11:20:29.859642 1437 raft_consensus.cc:348] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.860081 1437 raft_consensus.cc:374] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:29.860168 1437 raft_consensus.cc:729] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e850892f010f46239245e5e5d90b6b17, State: Initialized, Role: FOLLOWER
I20260929 11:20:29.860502 1436 ts_tablet_manager.cc:1428] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 11:20:29.860895 1437 consensus_queue.cc:260] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.861588 1438 raft_consensus.cc:348] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.861997 1438 raft_consensus.cc:374] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:29.862087 1438 raft_consensus.cc:729] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b9e3c9bd024f0d9ac62a5b1b4ffa31, State: Initialized, Role: FOLLOWER
I20260929 11:20:29.862603 1437 ts_tablet_manager.cc:1428] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 11:20:29.862746 1438 consensus_queue.cc:260] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.864534 1418 heartbeater.cc:503] Master 127.0.97.62:44903 was elected leader, sending a full tablet report...
I20260929 11:20:29.865272 1438 ts_tablet_manager.cc:1428] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 11:20:29.923167 1444 raft_consensus.cc:482] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:29.923435 1444 raft_consensus.cc:504] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.925585 1444 leader_election.cc:288] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e850892f010f46239245e5e5d90b6b17 (127.0.97.1:37617), 4fd8322d44484e6ba9b840ab90983cda (127.0.97.2:44303)
I20260929 11:20:29.932093 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.925986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56495 (local address 127.0.97.1:37617)
0929 11:20:29.926236 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.926243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.926262 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:20:29.926402 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.926414 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.926518 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.926618 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:29.927074 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.928108 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.930074 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.931255 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.931394 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.931492 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.931732 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.931853 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 11:20:29.932125 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.925777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37617 (local address 127.0.97.3:56495)
0929 11:20:29.926088 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.926110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.926236 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.926666 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.926672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.926703 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:29.926895 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.926903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.928237 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.928244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.929914 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.929928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.931402 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.931408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.931582 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.931605 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.931709 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.931842 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 11:20:29.932905 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.926070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44303 (local address 127.0.97.3:36467)
0929 11:20:29.926866 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:29.926883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:29.926976 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:29.927368 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:29.927375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:29.927401 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:29.927705 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.927725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.929198 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.929202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.930929 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:29.930944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.932121 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.932129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:29.932321 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.932350 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:20:29.932422 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 11:20:29.932559 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":449,"threads_started":1}
I20260929 11:20:29.933084 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.926071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36467 (local address 127.0.97.2:44303)
0929 11:20:29.926354 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.926364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.926978 (+ 614us) server_negotiation.cc:407] Connection header received
0929 11:20:29.927134 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.927137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.927235 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.927315 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:29.927994 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.929055 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.931152 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.931898 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.932057 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.932662 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.932718 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.932832 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 11:20:29.933084 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bc2d4d96184664b4b5f47712665a16" candidate_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e850892f010f46239245e5e5d90b6b17" is_pre_election: true
I20260929 11:20:29.933740 1106 raft_consensus.cc:2454] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b9e3c9bd024f0d9ac62a5b1b4ffa31 in term 0.
I20260929 11:20:29.933928 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bc2d4d96184664b4b5f47712665a16" candidate_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd8322d44484e6ba9b840ab90983cda" is_pre_election: true
I20260929 11:20:29.934454 1240 raft_consensus.cc:2454] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b9e3c9bd024f0d9ac62a5b1b4ffa31 in term 0.
I20260929 11:20:29.934506 1308 leader_election.cc:302] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b9e3c9bd024f0d9ac62a5b1b4ffa31, e850892f010f46239245e5e5d90b6b17; no voters:
I20260929 11:20:29.935299 1444 raft_consensus.cc:2781] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:29.935493 1444 raft_consensus.cc:482] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:29.935621 1444 raft_consensus.cc:3037] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:29.937845 1444 raft_consensus.cc:504] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.938417 1444 leader_election.cc:288] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [CANDIDATE]: Term 1 election: Requested vote from peers e850892f010f46239245e5e5d90b6b17 (127.0.97.1:37617), 4fd8322d44484e6ba9b840ab90983cda (127.0.97.2:44303)
I20260929 11:20:29.938884 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bc2d4d96184664b4b5f47712665a16" candidate_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e850892f010f46239245e5e5d90b6b17"
I20260929 11:20:29.938884 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bc2d4d96184664b4b5f47712665a16" candidate_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fd8322d44484e6ba9b840ab90983cda"
I20260929 11:20:29.939078 1106 raft_consensus.cc:3037] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:29.939078 1240 raft_consensus.cc:3037] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:29.942229 1106 raft_consensus.cc:2454] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b9e3c9bd024f0d9ac62a5b1b4ffa31 in term 1.
I20260929 11:20:29.942229 1240 raft_consensus.cc:2454] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b9e3c9bd024f0d9ac62a5b1b4ffa31 in term 1.
I20260929 11:20:29.942603 1308 leader_election.cc:302] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b9e3c9bd024f0d9ac62a5b1b4ffa31, e850892f010f46239245e5e5d90b6b17; no voters:
I20260929 11:20:29.942903 1444 raft_consensus.cc:2781] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:29.943275 1444 raft_consensus.cc:686] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [term 1 LEADER]: Becoming Leader. State: Replica: 42b9e3c9bd024f0d9ac62a5b1b4ffa31, State: Running, Role: LEADER
I20260929 11:20:29.943719 1444 consensus_queue.cc:237] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } }
I20260929 11:20:29.947866 960 catalog_manager.cc:5495] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42b9e3c9bd024f0d9ac62a5b1b4ffa31 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b9e3c9bd024f0d9ac62a5b1b4ffa31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e850892f010f46239245e5e5d90b6b17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd8322d44484e6ba9b840ab90983cda" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:20:29 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:29.977733 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:29.963354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.0.97.1:37617)
0929 11:20:29.963523 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:29.963528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:29.964014 (+ 486us) server_negotiation.cc:407] Connection header received
0929 11:20:29.964150 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:29.964153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:29.964241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:29.964323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:29.965828 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.966648 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.967835 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:29.968215 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:29.968312 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:29.972357 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:29.972371 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:29.972388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:29.972421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:29.975036 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.975684 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.975692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.975694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.975814 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:29.976172 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:29.976178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:29.976182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:29.976527 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:29.976876 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:29.977179 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 11:20:29.977397 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 11:20:29.978746 1419 tablet.cc:2369] T 07bc2d4d96184664b4b5f47712665a16 P 42b9e3c9bd024f0d9ac62a5b1b4ffa31: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680830, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 11:20:30.008775 1286 tablet.cc:2369] T 07bc2d4d96184664b4b5f47712665a16 P 4fd8322d44484e6ba9b840ab90983cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680830, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:30.030603 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:30.013147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.0.97.1:37617)
0929 11:20:30.013404 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:30.013413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:30.013927 (+ 514us) server_negotiation.cc:407] Connection header received
0929 11:20:30.014132 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:30.014139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:30.014254 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:30.014387 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:30.016664 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.017591 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.019428 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.019841 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.019941 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:30.024835 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:30.024845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:30.024853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:30.024900 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:30.028239 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.028897 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.028901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.028903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.029028 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.029386 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.029390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.029392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.029672 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:30.029854 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:30.030075 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 11:20:30.030246 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 11:20:30.037339 1152 tablet.cc:2369] T 07bc2d4d96184664b4b5f47712665a16 P e850892f010f46239245e5e5d90b6b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:30.073868 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:30.059825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.0.97.1:37617)
0929 11:20:30.060076 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:30.060083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:30.060331 (+ 248us) server_negotiation.cc:407] Connection header received
0929 11:20:30.060551 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:30.060555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:30.060684 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:30.060793 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:30.062089 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.062766 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.063962 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.064370 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.064495 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:30.068250 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:30.068255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:30.068258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:30.068287 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:30.070992 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.071808 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.071815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.071858 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.072001 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.072416 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.072421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.072424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.072769 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:30.073019 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:30.073287 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 11:20:30.073519 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 11:20:30.075500 1066 tablet_service.cc:2908] Rejecting scan request for tablet 07bc2d4d96184664b4b5f47712665a16: Uninitialized: clean time has not yet been initialized
I20260929 11:20:30.078608 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1023
I20260929 11:20:30.090111 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:30.079427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.0.97.1:37617)
0929 11:20:30.079593 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:30.079601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:30.079942 (+ 341us) server_negotiation.cc:407] Connection header received
0929 11:20:30.080017 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:30.080021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:30.080081 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:30.080152 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:30.081249 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.081889 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.082984 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.083402 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.083505 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:30.085505 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:30.085509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:30.085513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:30.085548 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:30.087890 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.088460 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.088464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.088466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.088581 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.089007 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.089019 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.089022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.089301 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:30.089515 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:30.089734 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 11:20:30.089887 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:20:30.090730 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:30.179204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1023
I20260929 11:20:30.196316 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1157
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:20:30.210718 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:30.197244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.97.2:44303)
0929 11:20:30.197427 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:30.197431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:30.197448 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:20:30.197503 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:30.197506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:30.197575 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:30.197657 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:30.198721 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.199440 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.200852 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.201674 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.201774 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:30.205774 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:30.205788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:30.205803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:30.205844 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:30.208356 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.209031 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.209038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.209040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.209150 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.209467 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.209470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.209472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.209815 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:30.210037 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:30.210228 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 11:20:30.210374 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:20:30.211375 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:30.298317 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1157
I20260929 11:20:30.314273 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1290
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:20:30.327987 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:30.314963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.0.97.3:33487)
0929 11:20:30.315114 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:30.315119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:30.315135 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:20:30.315285 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:30.315289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:30.315349 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:30.315432 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:30.316520 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.317300 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.318462 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:30.318883 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:30.318971 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:30.322759 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:30.322782 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:30.322799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:30.322827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:30.325625 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.326189 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.326197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.326200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.326321 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:30.326671 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:30.326674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:30.326676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:30.327004 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:30.327239 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:30.327473 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 11:20:30.327636 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:20:30.328771 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:30.416260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1290
I20260929 11:20:30.416306 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:30.415805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44303 (local address 127.0.97.3:46933)
0929 11:20:30.416003 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:30.416144 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:44303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:20:30.432569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 930
I20260929 11:20:30.433770 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:30.539702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 930
2026-09-29T11:20:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4021 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[1465](info): set up 2 sockets
Sep 29 11:20:30 dist-test-slave-srst krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260929 11:20:31.600550 388 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:31 dist-test-slave-srst krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:31.772456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45715
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:32.081956 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:32.082231 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:32.082379 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:32.089589 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:32.089690 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:20:32.089779 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:32.089888 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:32.089978 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:32.090090 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:32.099886 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:32.101188 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:32.102497 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:32.114753 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:32.114787 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:32.115031 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:32.115654 1478 server_base.cc:1034] running on GCE node
I20260929 11:20:32.116441 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:32.117862 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:32.119081 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790680832119007 us; error 51 us; skew 500 ppm
I20260929 11:20:32.119432 1478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:32 dist-test-slave-srst krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:32.122749 1478 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:32.124220 1478 webserver.cc:460] Webserver started at http://127.0.97.62:35085/ using document root <none> and password file <none>
I20260929 11:20:32.124995 1478 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:32.125119 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:32.125445 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:32.128232 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "8c379512bb874c62a6c43b7b5e73ad87"
format_stamp: "Formatted at 2026-09-29 11:20:32 on dist-test-slave-srst"
server_key: "6e9ae0be41fffb64b9bdce702ddca9fe"
server_key_iv: "c1ea307ad53eaec2a15385b8ba2ff020"
server_key_version: "encryptionkey@0"
I20260929 11:20:32.129009 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c379512bb874c62a6c43b7b5e73ad87"
format_stamp: "Formatted at 2026-09-29 11:20:32 on dist-test-slave-srst"
server_key: "6e9ae0be41fffb64b9bdce702ddca9fe"
server_key_iv: "c1ea307ad53eaec2a15385b8ba2ff020"
server_key_version: "encryptionkey@0"
I20260929 11:20:32.133657 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:20:32.137013 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:32.138254 1478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:20:32.138376 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "8c379512bb874c62a6c43b7b5e73ad87"
format_stamp: "Formatted at 2026-09-29 11:20:32 on dist-test-slave-srst"
server_key: "6e9ae0be41fffb64b9bdce702ddca9fe"
server_key_iv: "c1ea307ad53eaec2a15385b8ba2ff020"
server_key_version: "encryptionkey@0"
I20260929 11:20:32.138545 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:32.162365 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:32.166558 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:32.166797 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:32.187544 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45715
I20260929 11:20:32.187567 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45715 every 8 connection(s)
I20260929 11:20:32.188899 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:32.192746 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:32.196691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1478
I20260929 11:20:32.196918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:32.197546 388 external_mini_cluster.cc:1428] Setting key 44b0ca946bd5d14e9397e45a07f683d4
I20260929 11:20:32.204339 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: Bootstrap starting.
I20260929 11:20:32.207947 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:32.209251 1555 log.cc:824] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:32.212073 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: No bootstrap required, opened a new log
Sep 29 11:20:32 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:32.222230 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:32.199567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.97.62:45715)
0929 11:20:32.200375 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:32.200394 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:32.200443 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:20:32.202887 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:32.202922 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:32.203393 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:32.203795 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:32.205126 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.207151 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.208564 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.209655 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.209886 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:32.213905 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:32.213924 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:32.213942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:32.213988 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:32.217932 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:32.218592 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:32.218603 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:32.218621 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:32.218818 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:32.219151 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:32.219155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:32.219158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:32.219825 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:32.220138 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:32.220418 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 11:20:32.220879 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":353,"threads_started":1}
I20260929 11:20:32.222564 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c379512bb874c62a6c43b7b5e73ad87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45715 } }
I20260929 11:20:32.222929 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:32.223048 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c379512bb874c62a6c43b7b5e73ad87, State: Initialized, Role: FOLLOWER
I20260929 11:20:32.223603 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c379512bb874c62a6c43b7b5e73ad87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45715 } }
I20260929 11:20:32.223850 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:32.224001 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:32.224187 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:32.226491 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c379512bb874c62a6c43b7b5e73ad87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45715 } }
I20260929 11:20:32.226994 1555 leader_election.cc:302] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c379512bb874c62a6c43b7b5e73ad87; no voters:
I20260929 11:20:32.227533 1555 leader_election.cc:288] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:32.227727 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:32.228431 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [term 1 LEADER]: Becoming Leader. State: Replica: 8c379512bb874c62a6c43b7b5e73ad87, State: Running, Role: LEADER
I20260929 11:20:32.229050 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c379512bb874c62a6c43b7b5e73ad87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45715 } }
I20260929 11:20:32.229687 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:32.231876 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c379512bb874c62a6c43b7b5e73ad87. Latest consensus state: current_term: 1 leader_uuid: "8c379512bb874c62a6c43b7b5e73ad87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c379512bb874c62a6c43b7b5e73ad87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45715 } } }
I20260929 11:20:32.232235 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:32.232689 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c379512bb874c62a6c43b7b5e73ad87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c379512bb874c62a6c43b7b5e73ad87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45715 } } }
I20260929 11:20:32.232959 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:32.241537 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:32.247103 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:32.254854 1568 catalog_manager.cc:1347] Generated new cluster ID: c3268dbca0c249a28d04f3fdad0a4fe3
I20260929 11:20:32.254990 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:32.267695 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:32.269524 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:32.278052 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: Generated new TSK 0
I20260929 11:20:32.279042 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:32.327739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45715
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:32.656689 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:32.657003 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:32.657208 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:32.666028 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:32.666142 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:32.666224 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:20:32.666324 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:32.675926 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:32.677219 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:32.678488 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:32.690897 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:32.691021 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:32.691402 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:32.691906 1581 server_base.cc:1034] running on GCE node
I20260929 11:20:32.692530 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:32.693472 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:32.694710 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790680832694649 us; error 38 us; skew 500 ppm
I20260929 11:20:32.695026 1581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:32 dist-test-slave-srst krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:32.698005 1581 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:32.699397 1581 webserver.cc:460] Webserver started at http://127.0.97.1:36255/ using document root <none> and password file <none>
I20260929 11:20:32.700042 1581 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:32.700176 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:32.700486 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:32.703295 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "613759af2a6042fcbb3da8c182311081"
format_stamp: "Formatted at 2026-09-29 11:20:32 on dist-test-slave-srst"
server_key: "28b9eb7d2e345276c1e457250f164061"
server_key_iv: "8b1a229d345ea0db7ce596d53f0d9d91"
server_key_version: "encryptionkey@0"
I20260929 11:20:32.704054 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "613759af2a6042fcbb3da8c182311081"
format_stamp: "Formatted at 2026-09-29 11:20:32 on dist-test-slave-srst"
server_key: "28b9eb7d2e345276c1e457250f164061"
server_key_iv: "8b1a229d345ea0db7ce596d53f0d9d91"
server_key_version: "encryptionkey@0"
I20260929 11:20:32.708869 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260929 11:20:32.712119 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:32.713454 1581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:32.713649 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "613759af2a6042fcbb3da8c182311081"
format_stamp: "Formatted at 2026-09-29 11:20:32 on dist-test-slave-srst"
server_key: "28b9eb7d2e345276c1e457250f164061"
server_key_iv: "8b1a229d345ea0db7ce596d53f0d9d91"
server_key_version: "encryptionkey@0"
I20260929 11:20:32.713858 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:32.730930 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:32.734834 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:32.735107 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:32.737839 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:32.737977 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:32.738147 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:32.738224 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:20:32 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:32.766608 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:32.738643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42663 (local address 127.0.97.62:45715)
0929 11:20:32.741012 (+ 2369us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:32.741018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:32.741042 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:20:32.743117 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:32.743127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:32.743239 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:32.743383 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:32.745955 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.747275 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.750626 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.751421 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.751564 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:32.757787 (+ 6223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:32.757795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:32.757802 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:32.757852 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:32.761837 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:32.763076 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:32.763086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:32.763092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:32.763286 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:32.764017 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:32.764025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:32.764028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:32.764436 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:32.764636 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:32.765873 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0929 11:20:32.766158 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2232,"thread_start_us":2120,"threads_started":1}
I20260929 11:20:32.768743 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:32.739228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.97.1:42663)
0929 11:20:32.740371 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:32.740438 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:32.742722 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:32.743705 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:32.743716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:32.744543 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:32.745725 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:32.745756 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.747498 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.747506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:32.750414 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:32.750430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.751703 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.751714 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:32.752125 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:32.753213 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:32.753258 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:32.757422 (+ 4164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:32.762001 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:32.762017 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:32.762039 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:32.762885 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:32.763483 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:32.763507 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:32.763513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:32.763860 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:32.764635 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:32.764647 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:32.765259 (+ 612us) client_negotiation.cc:769] Sending connection context
0929 11:20:32.765731 (+ 472us) client_negotiation.cc:240] Negotiation successful
0929 11:20:32.766638 (+ 907us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":436,"threads_started":1}
I20260929 11:20:32.778750 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41211
I20260929 11:20:32.778846 1715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41211 every 8 connection(s)
I20260929 11:20:32.780449 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:32.782555 1717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45715
I20260929 11:20:32.783178 1717 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:32.784020 1717 heartbeater.cc:511] Master 127.0.97.62:45715 requested a full tablet report, sending...
I20260929 11:20:32.786339 1510 ts_manager.cc:194] Registered new tserver with Master: 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211)
I20260929 11:20:32.787153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1581
I20260929 11:20:32.787391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:32.788218 388 external_mini_cluster.cc:1428] Setting key 0293c157041e785cebce7d0f253c6a4b
I20260929 11:20:32.788733 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42663
I20260929 11:20:32.793255 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:32.779145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42501 (local address 127.0.97.62:45715)
0929 11:20:32.779551 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:32.779557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:32.779913 (+ 356us) server_negotiation.cc:407] Connection header received
0929 11:20:32.780055 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:32.780060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:32.780140 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:32.780236 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:32.781792 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.782717 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.784784 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.785285 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.785415 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:32.787187 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:32.787192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:32.787195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:32.787222 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:32.790533 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:32.791398 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:32.791403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:32.791407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:32.791543 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:32.791985 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:32.791989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:32.791991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:32.792219 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:32.792322 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:32.792890 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 11:20:32.793042 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260929 11:20:32.793783 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:32.779202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.97.1:42501)
0929 11:20:32.779813 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:32.779829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:32.779942 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:32.780264 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:32.780267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:32.780858 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:32.781379 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:32.781397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.782807 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.782811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:32.784649 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:32.784663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:32.785471 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:32.785479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:32.785652 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:32.786281 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:32.786296 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:32.787001 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:32.790676 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:32.790681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:32.790685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:32.791265 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:32.791679 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:32.791687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:32.791694 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:32.791867 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:32.792341 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:32.792350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:32.792583 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 11:20:32.792817 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 11:20:32.793187 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":364,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:32.828233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45715
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:33.134491 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:33.134807 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:33.134981 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:33.142720 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:33.142855 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:33.142989 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:20:33.143074 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:33.152444 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:33.153661 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:33.154947 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:33.167572 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:33.167670 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:33.167645 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:33.169064 1721 server_base.cc:1034] running on GCE node
I20260929 11:20:33.169628 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:33.170523 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:33.171782 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790680833171683 us; error 101 us; skew 500 ppm
I20260929 11:20:33.172125 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:33 dist-test-slave-srst krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:33.178491 1721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:33.179942 1721 webserver.cc:460] Webserver started at http://127.0.97.2:33923/ using document root <none> and password file <none>
I20260929 11:20:33.180878 1721 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:33.181000 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:33.181370 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:33.184228 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "9e882785921a49c48fb729bbbd15dae8"
format_stamp: "Formatted at 2026-09-29 11:20:33 on dist-test-slave-srst"
server_key: "9f8a2f0bc4f7aa678b9921961da58847"
server_key_iv: "a03f11a12c262e11bc9b5f53f3b8a8f6"
server_key_version: "encryptionkey@0"
I20260929 11:20:33.185010 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9e882785921a49c48fb729bbbd15dae8"
format_stamp: "Formatted at 2026-09-29 11:20:33 on dist-test-slave-srst"
server_key: "9f8a2f0bc4f7aa678b9921961da58847"
server_key_iv: "a03f11a12c262e11bc9b5f53f3b8a8f6"
server_key_version: "encryptionkey@0"
I20260929 11:20:33.190145 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 11:20:33.193747 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:33.195030 1721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:20:33.195189 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "9e882785921a49c48fb729bbbd15dae8"
format_stamp: "Formatted at 2026-09-29 11:20:33 on dist-test-slave-srst"
server_key: "9f8a2f0bc4f7aa678b9921961da58847"
server_key_iv: "a03f11a12c262e11bc9b5f53f3b8a8f6"
server_key_version: "encryptionkey@0"
I20260929 11:20:33.195397 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:33.242810 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:33.247129 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:33.247428 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:33.250205 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:33.250324 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:33.250504 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:33.250599 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:20:33 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:33.275807 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.250796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44695 (local address 127.0.97.62:45715)
0929 11:20:33.250996 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.251002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.252457 (+ 1455us) server_negotiation.cc:407] Connection header received
0929 11:20:33.254623 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.254633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.254726 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.254870 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:33.256942 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.257930 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.261477 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.262034 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.262181 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.267853 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:33.267860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:33.267868 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:33.267907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:33.272503 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.273485 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.273494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.273501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.273692 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.274159 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.274164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.274167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.274428 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:33.274577 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.275261 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.275474 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:20:33.276875 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.251418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.97.2:44695)
0929 11:20:33.252264 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.252313 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.254240 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.255159 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.255171 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.255924 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:33.256777 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.256795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.258139 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.258148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.261282 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.261298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.262313 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.262329 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.262753 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.263690 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:33.263732 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:33.267616 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:33.272711 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.272717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.272730 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.273329 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.273818 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.273821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.273823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.274023 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.274564 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:33.274572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:33.274900 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.275153 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.275665 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":395,"threads_started":1}
I20260929 11:20:33.299505 1810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.283011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.97.2:55905)
0929 11:20:33.283858 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.283885 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.284069 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.284391 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.284400 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.285242 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:33.285783 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.285799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.287007 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.287016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.289650 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.289670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.290882 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.290887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.290995 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.291649 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:33.291683 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:33.293125 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:33.296526 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.296530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.296534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.297233 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.297590 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.297595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.297599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.297724 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.298407 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:33.298412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:33.298620 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.298807 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.299157 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":381,"threads_started":1}
I20260929 11:20:33.299837 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.283041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55905 (local address 127.0.97.62:45715)
0929 11:20:33.283274 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.283283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.283980 (+ 697us) server_negotiation.cc:407] Connection header received
0929 11:20:33.284188 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.284191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.284245 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.284320 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:33.285957 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.286876 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.289884 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.290708 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.290877 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.293297 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:33.293301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:33.293304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:33.293332 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:33.296401 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.297359 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.297364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.297368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.297481 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.297840 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.297845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.297848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.298306 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:33.298442 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.298979 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.299288 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:20:33.307629 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43093
I20260929 11:20:33.307650 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43093 every 8 connection(s)
I20260929 11:20:33.308952 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:33.310999 1856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45715
I20260929 11:20:33.311178 1856 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:33.311707 1856 heartbeater.cc:511] Master 127.0.97.62:45715 requested a full tablet report, sending...
I20260929 11:20:33.313021 1509 ts_manager.cc:194] Registered new tserver with Master: 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093)
I20260929 11:20:33.313800 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44695
I20260929 11:20:33.314386 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1721
I20260929 11:20:33.314602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:33.315039 388 external_mini_cluster.cc:1428] Setting key b5a00521eedd804da1b30bbc378fa26d
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:33.355006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45715
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:33.663667 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:33.663968 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:33.664119 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:33.671485 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:33.671619 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:33.671777 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:20:33.671914 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:33.681885 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:33.683094 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:33.684350 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:33.697079 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:33.697331 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:33.697602 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:33.697705 1860 server_base.cc:1034] running on GCE node
I20260929 11:20:33.698406 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:33.699308 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:33.700619 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790680833700473 us; error 60 us; skew 500 ppm
I20260929 11:20:33.700920 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:33 dist-test-slave-srst krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:33.703817 1860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:33.705284 1860 webserver.cc:460] Webserver started at http://127.0.97.3:34953/ using document root <none> and password file <none>
I20260929 11:20:33.705945 1860 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:33.706071 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:33.706386 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:33.709282 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "50d5e450300d4f9da698fd859d8d356b"
format_stamp: "Formatted at 2026-09-29 11:20:33 on dist-test-slave-srst"
server_key: "d77cc9ab97c0efd610aacb6c82d2d8bd"
server_key_iv: "0f8cd754a119dbe0839b35195598bd4f"
server_key_version: "encryptionkey@0"
I20260929 11:20:33.710104 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "50d5e450300d4f9da698fd859d8d356b"
format_stamp: "Formatted at 2026-09-29 11:20:33 on dist-test-slave-srst"
server_key: "d77cc9ab97c0efd610aacb6c82d2d8bd"
server_key_iv: "0f8cd754a119dbe0839b35195598bd4f"
server_key_version: "encryptionkey@0"
I20260929 11:20:33.714950 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:20:33.718405 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:33.719719 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:20:33.719883 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "50d5e450300d4f9da698fd859d8d356b"
format_stamp: "Formatted at 2026-09-29 11:20:33 on dist-test-slave-srst"
server_key: "d77cc9ab97c0efd610aacb6c82d2d8bd"
server_key_iv: "0f8cd754a119dbe0839b35195598bd4f"
server_key_version: "encryptionkey@0"
I20260929 11:20:33.720093 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:33.735533 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:33.740100 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:33.740365 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:33.743254 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:33.743351 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:33.743480 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:33.743554 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:20:33 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:33.765940 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.743896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53795 (local address 127.0.97.62:45715)
0929 11:20:33.744080 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.744085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.745597 (+ 1512us) server_negotiation.cc:407] Connection header received
0929 11:20:33.747677 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.747683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.747766 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.747890 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:33.750109 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.751209 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.753919 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.754384 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.754519 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.758789 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:33.758798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:33.758803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:33.758856 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:33.762327 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.763418 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.763422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.763424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.763546 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.764083 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.764086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.764088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.764358 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:33.764462 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.765509 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.765699 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:20:33.768103 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.744445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.97.3:53795)
0929 11:20:33.745385 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.745456 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.747363 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.748119 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.748127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.748886 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:33.749842 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.749872 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.751378 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.751412 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.753774 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.753786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.754616 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.754624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.754826 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.755651 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:33.755673 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:33.758528 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:33.762473 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.762487 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.762510 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.763261 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.763655 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.763662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.763668 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.763971 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.764482 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:33.764494 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:33.765052 (+ 558us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.765411 (+ 359us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.766134 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":444,"threads_started":1}
I20260929 11:20:33.792104 1717 heartbeater.cc:503] Master 127.0.97.62:45715 was elected leader, sending a full tablet report...
I20260929 11:20:33.792614 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46779
I20260929 11:20:33.792656 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46779 every 8 connection(s)
I20260929 11:20:33.794109 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:33.794872 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.773935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45635 (local address 127.0.97.62:45715)
0929 11:20:33.774169 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.774177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.775331 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 11:20:33.775523 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.775531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.775621 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.775748 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:33.776710 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.778024 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.784697 (+ 6673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.785480 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.785649 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.787982 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:33.787990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:33.787997 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:33.788041 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:33.791688 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.792741 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.792746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.792749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.792907 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.793302 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.793310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.793313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.793629 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:33.793759 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.794315 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.794523 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:20:33.795004 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.774353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.97.3:45635)
0929 11:20:33.775172 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.775193 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.775384 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.775774 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.775781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.776255 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:33.776528 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.776536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.778151 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.778156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.779472 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.779483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.785924 (+ 6441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.785933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.786201 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.787049 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:33.787071 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:33.787784 (+ 713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:33.791846 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.791852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.791858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.792548 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.793016 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:33.793022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:33.793025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:33.793185 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:33.793837 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:33.793844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:33.794039 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.794203 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.794523 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":450,"threads_started":1}
I20260929 11:20:33.796720 1995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45715
I20260929 11:20:33.796940 1995 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:33.797735 1995 heartbeater.cc:511] Master 127.0.97.62:45715 requested a full tablet report, sending...
I20260929 11:20:33.798966 1509 ts_manager.cc:194] Registered new tserver with Master: 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779)
I20260929 11:20:33.799827 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53795
I20260929 11:20:33.801036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 1860
I20260929 11:20:33.801285 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:33.801710 388 external_mini_cluster.cc:1428] Setting key fd56e381bdeac5fc3a80e146a8f8f297
I20260929 11:20:33.804358 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:20:33 dist-test-slave-srst krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:20:33 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:33.847962 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.834383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.0.97.62:45715)
0929 11:20:33.834606 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.834613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.834858 (+ 245us) server_negotiation.cc:407] Connection header received
0929 11:20:33.835077 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.835081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.835142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.835253 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:33.836681 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.837416 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.838774 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.839255 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.839389 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.843355 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:33.843359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:33.843363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:33.843395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:33.845771 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.846336 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.846340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.846343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.846450 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:33.846763 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:33.846766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:33.846768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:33.847013 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:33.847188 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.847408 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.847559 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:20:33.851765 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:33.858479 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:33.884442 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.870236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.0.97.2:43093)
0929 11:20:33.872143 (+ 1907us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.872153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.872183 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:20:33.872286 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.872296 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.872548 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.872789 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:33.874997 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.877927 (+ 2930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.881817 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.883234 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.883373 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.883611 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.883992 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.884156 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":4796,"threads_started":1}
I20260929 11:20:33.884491 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.870046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43093 (local address 127.0.0.1:41002)
0929 11:20:33.870739 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.870774 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.870934 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.874198 (+ 3264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.874208 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.874245 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:33.874751 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.874776 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.878098 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.878106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.881621 (+ 3515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.881635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.883440 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.883447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.883741 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.883775 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.883930 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.884135 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":354,"threads_started":1}
I20260929 11:20:33.888350 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 883af90e8b304ef6beafdd9b55523c0a (DEFAULT_TABLE table=test-table [id=b5e6dc03e10348a8a1a94b29e3e4f678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:33.889843 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883af90e8b304ef6beafdd9b55523c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:33.890136 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.871173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41211 (local address 127.0.0.1:59006)
0929 11:20:33.880716 (+ 9543us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.880742 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.880879 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.882909 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.882913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.882935 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:33.883358 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.883373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.885465 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.885472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.887602 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.887615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.889411 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.889415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.889517 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.889538 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.889664 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.889790 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9361,"thread_start_us":10747,"threads_started":1}
I20260929 11:20:33.890471 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.871478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.0.97.1:41211)
0929 11:20:33.882455 (+ 10977us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.882461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.882482 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:20:33.882551 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.882556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.882727 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.882911 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:33.883613 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.885341 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.887766 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.889011 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.889161 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.889308 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.889710 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.889871 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10835,"thread_start_us":10716,"threads_started":1}
I20260929 11:20:33.891175 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.871985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46779 (local address 127.0.0.1:58594)
0929 11:20:33.882140 (+ 10155us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:33.882158 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:33.882266 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:33.883141 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:33.883147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:33.883174 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:33.883673 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.883684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.886413 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.886418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.888255 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:33.888268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.890438 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.890443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:33.890607 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.890637 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:20:33.890764 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 11:20:33.890885 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10048,"thread_start_us":8374,"threads_started":1}
I20260929 11:20:33.891374 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:33.872570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.0.97.3:46779)
0929 11:20:33.881565 (+ 8995us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:33.881578 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:33.882371 (+ 793us) server_negotiation.cc:407] Connection header received
0929 11:20:33.882471 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:33.882478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:33.882746 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:33.882925 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:33.883896 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.885470 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.888424 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:33.889842 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:33.889962 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:33.890183 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:33.890827 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 11:20:33.891020 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8756,"thread_start_us":8555,"threads_started":1}
I20260929 11:20:33.895043 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 883af90e8b304ef6beafdd9b55523c0a (DEFAULT_TABLE table=test-table [id=b5e6dc03e10348a8a1a94b29e3e4f678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:33.895002 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 883af90e8b304ef6beafdd9b55523c0a (DEFAULT_TABLE table=test-table [id=b5e6dc03e10348a8a1a94b29e3e4f678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:33.896102 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883af90e8b304ef6beafdd9b55523c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:33.896302 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883af90e8b304ef6beafdd9b55523c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:33.904268 2014 tablet_bootstrap.cc:492] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Bootstrap starting.
I20260929 11:20:33.907665 2014 tablet_bootstrap.cc:654] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:33.908576 2015 tablet_bootstrap.cc:492] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Bootstrap starting.
I20260929 11:20:33.909145 2014 log.cc:824] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:33.911713 2015 tablet_bootstrap.cc:654] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:33.911877 2014 tablet_bootstrap.cc:492] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: No bootstrap required, opened a new log
I20260929 11:20:33.912128 2014 ts_tablet_manager.cc:1397] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 11:20:33.912791 2015 log.cc:824] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:33.913437 2017 tablet_bootstrap.cc:492] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Bootstrap starting.
I20260929 11:20:33.915570 2015 tablet_bootstrap.cc:492] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: No bootstrap required, opened a new log
I20260929 11:20:33.915827 2015 ts_tablet_manager.cc:1397] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 11:20:33.916680 2017 tablet_bootstrap.cc:654] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:33.918411 2017 log.cc:824] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:33.920101 2014 raft_consensus.cc:348] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:33.920425 2014 raft_consensus.cc:374] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:33.920542 2014 raft_consensus.cc:729] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e882785921a49c48fb729bbbd15dae8, State: Initialized, Role: FOLLOWER
I20260929 11:20:33.921192 2017 tablet_bootstrap.cc:492] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: No bootstrap required, opened a new log
I20260929 11:20:33.921136 2014 consensus_queue.cc:260] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:33.921444 2017 ts_tablet_manager.cc:1397] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 11:20:33.926208 2015 raft_consensus.cc:348] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:33.926591 2015 raft_consensus.cc:374] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:33.926685 2015 raft_consensus.cc:729] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613759af2a6042fcbb3da8c182311081, State: Initialized, Role: FOLLOWER
I20260929 11:20:33.927265 2015 consensus_queue.cc:260] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:33.930179 2015 ts_tablet_manager.cc:1428] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260929 11:20:33.933439 1856 heartbeater.cc:503] Master 127.0.97.62:45715 was elected leader, sending a full tablet report...
I20260929 11:20:33.933890 2014 ts_tablet_manager.cc:1428] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Time spent starting tablet: real 0.022s user 0.010s sys 0.012s
I20260929 11:20:33.934831 2017 raft_consensus.cc:348] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:33.935226 2017 raft_consensus.cc:374] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:33.935341 2017 raft_consensus.cc:729] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d5e450300d4f9da698fd859d8d356b, State: Initialized, Role: FOLLOWER
I20260929 11:20:33.935891 2017 consensus_queue.cc:260] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:33.938081 1995 heartbeater.cc:503] Master 127.0.97.62:45715 was elected leader, sending a full tablet report...
I20260929 11:20:33.938612 2017 ts_tablet_manager.cc:1428] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260929 11:20:34.032878 1718 tablet.cc:2369] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:20:34.045960 1996 tablet.cc:2369] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:20:34.060871 1857 tablet.cc:2369] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:20:34.073822 2022 raft_consensus.cc:482] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:34.074162 2022 raft_consensus.cc:504] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.075892 2022 leader_election.cc:288] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093), 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211)
I20260929 11:20:34.082057 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.076212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43093 (local address 127.0.97.3:55849)
0929 11:20:34.076409 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.076426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.076525 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.076923 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.076929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.076955 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:34.077211 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.077243 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.078541 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.078544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.080367 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.080380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.081555 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.081558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.081682 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.081705 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.081778 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.081863 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:20:34.082185 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.076222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55849 (local address 127.0.97.2:43093)
0929 11:20:34.076462 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.076468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.076493 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:20:34.076689 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.076694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.076780 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.076869 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:34.077376 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.078409 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.080525 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.081414 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.081542 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.081652 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.081820 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.081943 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:20:34.083232 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.076787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60195 (local address 127.0.97.1:41211)
0929 11:20:34.077014 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.077023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.077536 (+ 513us) server_negotiation.cc:407] Connection header received
0929 11:20:34.077635 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.077642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.077731 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.077836 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:34.078259 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.079071 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.081229 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.082432 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.082565 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.082663 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.082973 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.083057 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:20:34.083078 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883af90e8b304ef6beafdd9b55523c0a" candidate_uuid: "50d5e450300d4f9da698fd859d8d356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e882785921a49c48fb729bbbd15dae8" is_pre_election: true
I20260929 11:20:34.083365 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.076675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41211 (local address 127.0.97.3:60195)
0929 11:20:34.077449 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.077461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.077544 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.077842 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.077845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.077860 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:34.078130 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.078136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.079206 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.079211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.081071 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.081081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.082638 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.082647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.082818 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.082855 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.082952 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.083065 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":469,"threads_started":1}
I20260929 11:20:34.083653 1811 raft_consensus.cc:2454] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d5e450300d4f9da698fd859d8d356b in term 0.
I20260929 11:20:34.084357 1877 leader_election.cc:302] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d5e450300d4f9da698fd859d8d356b, 9e882785921a49c48fb729bbbd15dae8; no voters:
I20260929 11:20:34.084695 2022 raft_consensus.cc:2781] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:34.084802 2022 raft_consensus.cc:482] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:34.084918 2022 raft_consensus.cc:3037] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.084312 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883af90e8b304ef6beafdd9b55523c0a" candidate_uuid: "50d5e450300d4f9da698fd859d8d356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613759af2a6042fcbb3da8c182311081" is_pre_election: true
I20260929 11:20:34.085246 1667 raft_consensus.cc:2454] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d5e450300d4f9da698fd859d8d356b in term 0.
I20260929 11:20:34.087234 2022 raft_consensus.cc:504] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.087807 2022 leader_election.cc:288] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [CANDIDATE]: Term 1 election: Requested vote from peers 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093), 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211)
I20260929 11:20:34.088203 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883af90e8b304ef6beafdd9b55523c0a" candidate_uuid: "50d5e450300d4f9da698fd859d8d356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e882785921a49c48fb729bbbd15dae8"
I20260929 11:20:34.088253 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883af90e8b304ef6beafdd9b55523c0a" candidate_uuid: "50d5e450300d4f9da698fd859d8d356b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613759af2a6042fcbb3da8c182311081"
I20260929 11:20:34.088414 1667 raft_consensus.cc:3037] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.088418 1811 raft_consensus.cc:3037] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.091432 1667 raft_consensus.cc:2454] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d5e450300d4f9da698fd859d8d356b in term 1.
I20260929 11:20:34.091434 1811 raft_consensus.cc:2454] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d5e450300d4f9da698fd859d8d356b in term 1.
I20260929 11:20:34.091781 1877 leader_election.cc:302] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d5e450300d4f9da698fd859d8d356b, 613759af2a6042fcbb3da8c182311081; no voters:
I20260929 11:20:34.092058 2022 raft_consensus.cc:2781] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:34.092443 2022 raft_consensus.cc:686] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 1 LEADER]: Becoming Leader. State: Replica: 50d5e450300d4f9da698fd859d8d356b, State: Running, Role: LEADER
I20260929 11:20:34.092895 2022 consensus_queue.cc:237] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.097254 1509 catalog_manager.cc:5495] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d5e450300d4f9da698fd859d8d356b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "50d5e450300d4f9da698fd859d8d356b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } health_report { overall_health: HEALTHY } } }
I20260929 11:20:34.118893 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.113574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.0.97.3:46779)
0929 11:20:34.113739 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.113743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.113761 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:20:34.114023 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.114026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.114105 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.114242 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:34.114699 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.115536 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.116917 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.117358 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.117449 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.117919 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.118418 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.118520 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.118634 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.118699 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:20:34.126885 1667 raft_consensus.cc:1261] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Refusing update from remote peer 50d5e450300d4f9da698fd859d8d356b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:34.126885 1811 raft_consensus.cc:1261] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Refusing update from remote peer 50d5e450300d4f9da698fd859d8d356b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:34.127795 2024 consensus_queue.cc:1035] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [LEADER]: Connected to new peer: Peer: permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:34.128149 2022 consensus_queue.cc:1035] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:34.139988 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334628696573382656 to 7334628696447393792. Current Snapshot: MvccSnapshot[applied={T|T < 7334628696573382656}]
I20260929 11:20:34.157928 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:34.158210 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:34.161764 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: Sending DeleteTablet for 3 replicas of tablet 883af90e8b304ef6beafdd9b55523c0a
I20260929 11:20:34.162971 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 883af90e8b304ef6beafdd9b55523c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58594
I20260929 11:20:34.163146 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 883af90e8b304ef6beafdd9b55523c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59006
I20260929 11:20:34.163218 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 883af90e8b304ef6beafdd9b55523c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41002
I20260929 11:20:34.167843 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:34.168756 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:34.174245 2037 tablet_replica.cc:330] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: stopping tablet replica
I20260929 11:20:34.174817 2039 tablet_replica.cc:330] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: stopping tablet replica
I20260929 11:20:34.175379 2039 raft_consensus.cc:2229] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:34.175865 2037 raft_consensus.cc:2229] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:20:34.176391 2039 raft_consensus.cc:2258] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:34.177064 2037 raft_consensus.cc:2258] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:34.177923 2038 tablet_replica.cc:330] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: stopping tablet replica
I20260929 11:20:34.178537 1786 tablet_service.cc:1467] Processing CreateTablet for tablet b54c0d8b47ee46a7878c37231525fe51 (DEFAULT_TABLE table=test-table [id=a6c068c9315d4447b1e6b66285669b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:34.179013 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54c0d8b47ee46a7878c37231525fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:34.179338 1925 tablet_service.cc:1467] Processing CreateTablet for tablet b54c0d8b47ee46a7878c37231525fe51 (DEFAULT_TABLE table=test-table [id=a6c068c9315d4447b1e6b66285669b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:34.178740 1647 tablet_service.cc:1467] Processing CreateTablet for tablet b54c0d8b47ee46a7878c37231525fe51 (DEFAULT_TABLE table=test-table [id=a6c068c9315d4447b1e6b66285669b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:34.179864 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54c0d8b47ee46a7878c37231525fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:34.180619 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54c0d8b47ee46a7878c37231525fe51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:34.185451 2038 raft_consensus.cc:2229] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:34.186012 2038 raft_consensus.cc:2258] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:34.187927 2038 ts_tablet_manager.cc:1905] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:34.189013 2037 ts_tablet_manager.cc:1905] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:34.191577 2015 tablet_bootstrap.cc:492] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Bootstrap starting.
I20260929 11:20:34.191974 2039 ts_tablet_manager.cc:1905] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:34.193779 2015 tablet_bootstrap.cc:654] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:34.193972 2014 tablet_bootstrap.cc:492] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Bootstrap starting.
I20260929 11:20:34.195556 2014 tablet_bootstrap.cc:654] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:34.199532 2039 ts_tablet_manager.cc:1918] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:20:34.199802 2039 log.cc:1196] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/883af90e8b304ef6beafdd9b55523c0a
I20260929 11:20:34.200623 2017 tablet_bootstrap.cc:492] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Bootstrap starting.
I20260929 11:20:34.202842 2017 tablet_bootstrap.cc:654] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:34.206429 2039 ts_tablet_manager.cc:1939] T 883af90e8b304ef6beafdd9b55523c0a P 9e882785921a49c48fb729bbbd15dae8: Deleting consensus metadata
I20260929 11:20:34.208528 1495 catalog_manager.cc:4841] TS 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093): tablet 883af90e8b304ef6beafdd9b55523c0a (table test-table [id=b5e6dc03e10348a8a1a94b29e3e4f678]) successfully deleted
I20260929 11:20:34.210237 2014 tablet_bootstrap.cc:492] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: No bootstrap required, opened a new log
I20260929 11:20:34.210379 2014 ts_tablet_manager.cc:1397] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260929 11:20:34.211467 2014 raft_consensus.cc:348] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.211741 2014 raft_consensus.cc:374] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:34.211843 2014 raft_consensus.cc:729] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e882785921a49c48fb729bbbd15dae8, State: Initialized, Role: FOLLOWER
I20260929 11:20:34.212150 2014 consensus_queue.cc:260] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.214152 2037 ts_tablet_manager.cc:1918] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:20:34.214771 2015 tablet_bootstrap.cc:492] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: No bootstrap required, opened a new log
I20260929 11:20:34.214959 2015 ts_tablet_manager.cc:1397] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.006s
I20260929 11:20:34.215008 2037 log.cc:1196] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/883af90e8b304ef6beafdd9b55523c0a
I20260929 11:20:34.215437 2014 ts_tablet_manager.cc:1428] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260929 11:20:34.215842 2015 raft_consensus.cc:348] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.216107 2015 raft_consensus.cc:374] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:34.216243 2015 raft_consensus.cc:729] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613759af2a6042fcbb3da8c182311081, State: Initialized, Role: FOLLOWER
I20260929 11:20:34.216610 2015 consensus_queue.cc:260] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.217947 2037 ts_tablet_manager.cc:1939] T 883af90e8b304ef6beafdd9b55523c0a P 50d5e450300d4f9da698fd859d8d356b: Deleting consensus metadata
I20260929 11:20:34.218230 2038 ts_tablet_manager.cc:1918] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:20:34.218401 2038 log.cc:1196] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/883af90e8b304ef6beafdd9b55523c0a
I20260929 11:20:34.219383 2038 ts_tablet_manager.cc:1939] T 883af90e8b304ef6beafdd9b55523c0a P 613759af2a6042fcbb3da8c182311081: Deleting consensus metadata
I20260929 11:20:34.219673 2015 ts_tablet_manager.cc:1428] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 11:20:34.221041 1495 catalog_manager.cc:4841] TS 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211): tablet 883af90e8b304ef6beafdd9b55523c0a (table test-table [id=b5e6dc03e10348a8a1a94b29e3e4f678]) successfully deleted
I20260929 11:20:34.222467 1497 catalog_manager.cc:4841] TS 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779): tablet 883af90e8b304ef6beafdd9b55523c0a (table test-table [id=b5e6dc03e10348a8a1a94b29e3e4f678]) successfully deleted
I20260929 11:20:34.223168 2017 tablet_bootstrap.cc:492] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: No bootstrap required, opened a new log
I20260929 11:20:34.223312 2017 ts_tablet_manager.cc:1397] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20260929 11:20:34.224049 2017 raft_consensus.cc:348] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.224275 2017 raft_consensus.cc:374] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:34.224359 2017 raft_consensus.cc:729] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d5e450300d4f9da698fd859d8d356b, State: Initialized, Role: FOLLOWER
I20260929 11:20:34.224617 2017 consensus_queue.cc:260] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.225273 2017 ts_tablet_manager.cc:1428] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:20:34.520821 2021 raft_consensus.cc:482] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:34.521085 2021 raft_consensus.cc:504] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.523095 2021 leader_election.cc:288] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093), 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779)
I20260929 11:20:34.532022 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.524063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48181 (local address 127.0.97.2:43093)
0929 11:20:34.524237 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.524243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.525301 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 11:20:34.525565 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.525571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.525644 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.525745 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:34.526421 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.527236 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.529648 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.530952 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.531088 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.531195 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.531603 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.531726 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:20:34.532169 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.523432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43093 (local address 127.0.97.1:48181)
0929 11:20:34.525175 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.525189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.525398 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.525800 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.525807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.525837 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:34.526281 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.526291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.527379 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.527384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.529473 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.529494 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.531175 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.531181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.531419 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.531469 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.531579 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.531716 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"mutex_wait_us":3,"thread_start_us":1458,"threads_started":1}
I20260929 11:20:34.532748 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.523668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33131 (local address 127.0.97.3:46779)
0929 11:20:34.523864 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.523872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.525019 (+ 1147us) server_negotiation.cc:407] Connection header received
0929 11:20:34.525329 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.525336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.525417 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.525512 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:34.526200 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.527851 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.530062 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.531090 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.531215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.531308 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.531915 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.532052 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:20:34.532924 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54c0d8b47ee46a7878c37231525fe51" candidate_uuid: "613759af2a6042fcbb3da8c182311081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e882785921a49c48fb729bbbd15dae8" is_pre_election: true
I20260929 11:20:34.533135 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.523381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46779 (local address 127.0.97.1:33131)
0929 11:20:34.524891 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.524912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.525141 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.525553 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.525559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.525583 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:34.525993 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.526010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.528011 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.528015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.529899 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.529909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.531421 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.531428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.531697 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.531743 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.532653 (+ 910us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.532780 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":53,"thread_start_us":1082,"threads_started":1}
I20260929 11:20:34.533195 1811 raft_consensus.cc:2454] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613759af2a6042fcbb3da8c182311081 in term 0.
I20260929 11:20:34.533689 1598 leader_election.cc:302] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613759af2a6042fcbb3da8c182311081, 9e882785921a49c48fb729bbbd15dae8; no voters:
I20260929 11:20:34.534255 2021 raft_consensus.cc:2781] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:34.534188 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54c0d8b47ee46a7878c37231525fe51" candidate_uuid: "613759af2a6042fcbb3da8c182311081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d5e450300d4f9da698fd859d8d356b" is_pre_election: true
I20260929 11:20:34.534390 2021 raft_consensus.cc:482] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:34.534483 2021 raft_consensus.cc:3037] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.534617 1950 raft_consensus.cc:2454] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613759af2a6042fcbb3da8c182311081 in term 0.
I20260929 11:20:34.538182 2021 raft_consensus.cc:504] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.538906 2021 leader_election.cc:288] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [CANDIDATE]: Term 1 election: Requested vote from peers 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093), 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779)
I20260929 11:20:34.539386 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54c0d8b47ee46a7878c37231525fe51" candidate_uuid: "613759af2a6042fcbb3da8c182311081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d5e450300d4f9da698fd859d8d356b"
I20260929 11:20:34.539429 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54c0d8b47ee46a7878c37231525fe51" candidate_uuid: "613759af2a6042fcbb3da8c182311081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e882785921a49c48fb729bbbd15dae8"
I20260929 11:20:34.539597 1950 raft_consensus.cc:3037] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.539642 1811 raft_consensus.cc:3037] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.541966 1950 raft_consensus.cc:2454] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613759af2a6042fcbb3da8c182311081 in term 1.
I20260929 11:20:34.542383 1600 leader_election.cc:302] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d5e450300d4f9da698fd859d8d356b, 613759af2a6042fcbb3da8c182311081; no voters:
I20260929 11:20:34.542654 2021 raft_consensus.cc:2781] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:34.543147 1811 raft_consensus.cc:2454] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613759af2a6042fcbb3da8c182311081 in term 1.
I20260929 11:20:34.543174 2021 raft_consensus.cc:686] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 1 LEADER]: Becoming Leader. State: Replica: 613759af2a6042fcbb3da8c182311081, State: Running, Role: LEADER
I20260929 11:20:34.543777 2021 consensus_queue.cc:237] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.547423 1509 catalog_manager.cc:5495] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 reported cstate change: term changed from 0 to 1, leader changed from <none> to 613759af2a6042fcbb3da8c182311081 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "613759af2a6042fcbb3da8c182311081" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:20:34 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:34.680714 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.666776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45715 (local address 127.0.0.1:57396)
0929 11:20:34.667576 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.667596 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.667724 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.668024 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.668027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.668717 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:34.668905 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.668913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.669986 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.669991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.671280 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.671292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.672073 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.672078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.672246 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.672939 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:34.672958 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:34.675300 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:34.678259 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:34.678269 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:34.678277 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:34.678679 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:34.679115 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:34.679120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:34.679123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:34.679284 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:34.679778 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:34.679785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:34.679962 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.680143 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.680350 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":483,"threads_started":1}
I20260929 11:20:34.680799 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.666798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.0.97.62:45715)
0929 11:20:34.667599 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.667603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.667673 (+ 70us) server_negotiation.cc:407] Connection header received
0929 11:20:34.667827 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.667831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.667887 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.667979 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:34.669043 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.669790 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.671491 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.671895 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.672039 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.675496 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:34.675500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:34.675503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:34.675528 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:34.678112 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:34.678828 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:34.678835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:34.678838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:34.679004 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:34.679381 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:34.679384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:34.679386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:34.679639 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:34.679765 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.680276 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.680432 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":542,"threads_started":1}
I20260929 11:20:34.685689 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57396:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:20:34.686657 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:34.692813 1647 tablet_service.cc:1467] Processing CreateTablet for tablet e50cecaa874e4b48a3ce224e7f4934c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6817a748dc654732add2c9fcb3593f9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:20:34.692981 1925 tablet_service.cc:1467] Processing CreateTablet for tablet e50cecaa874e4b48a3ce224e7f4934c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6817a748dc654732add2c9fcb3593f9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:20:34.692982 1786 tablet_service.cc:1467] Processing CreateTablet for tablet e50cecaa874e4b48a3ce224e7f4934c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6817a748dc654732add2c9fcb3593f9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:20:34.693352 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50cecaa874e4b48a3ce224e7f4934c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:34.693506 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50cecaa874e4b48a3ce224e7f4934c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:34.693495 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50cecaa874e4b48a3ce224e7f4934c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:34.698725 2015 tablet_bootstrap.cc:492] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: Bootstrap starting.
I20260929 11:20:34.700296 2015 tablet_bootstrap.cc:654] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:34.700822 2014 tablet_bootstrap.cc:492] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: Bootstrap starting.
I20260929 11:20:34.701426 2017 tablet_bootstrap.cc:492] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: Bootstrap starting.
I20260929 11:20:34.702378 2014 tablet_bootstrap.cc:654] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:34.703159 2015 tablet_bootstrap.cc:492] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: No bootstrap required, opened a new log
I20260929 11:20:34.703341 2015 ts_tablet_manager.cc:1397] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 11:20:34.703418 2017 tablet_bootstrap.cc:654] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:34.704134 2014 tablet_bootstrap.cc:492] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: No bootstrap required, opened a new log
I20260929 11:20:34.704047 2015 raft_consensus.cc:348] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.704327 2015 raft_consensus.cc:374] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:34.704363 2014 ts_tablet_manager.cc:1397] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 11:20:34.704403 2015 raft_consensus.cc:729] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613759af2a6042fcbb3da8c182311081, State: Initialized, Role: FOLLOWER
I20260929 11:20:34.704663 2015 consensus_queue.cc:260] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.705209 2015 ts_tablet_manager.cc:1428] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:34.705112 2014 raft_consensus.cc:348] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.705080 2021 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.705317 2014 raft_consensus.cc:374] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:34.705358 2021 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.705384 2014 raft_consensus.cc:729] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e882785921a49c48fb729bbbd15dae8, State: Initialized, Role: FOLLOWER
I20260929 11:20:34.705638 2014 consensus_queue.cc:260] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.706072 2020 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.706393 2020 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.706833 2017 tablet_bootstrap.cc:492] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: No bootstrap required, opened a new log
I20260929 11:20:34.706835 2014 ts_tablet_manager.cc:1428] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:34.707062 2017 ts_tablet_manager.cc:1397] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 11:20:34.707752 2017 raft_consensus.cc:348] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.707984 2017 raft_consensus.cc:374] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:34.708067 2017 raft_consensus.cc:729] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d5e450300d4f9da698fd859d8d356b, State: Initialized, Role: FOLLOWER
I20260929 11:20:34.708287 2017 consensus_queue.cc:260] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.708853 2017 ts_tablet_manager.cc:1428] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:34.708743 2034 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.709012 2034 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.796027 2020 raft_consensus.cc:482] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:34.796252 2020 raft_consensus.cc:504] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.797753 2020 leader_election.cc:288] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211), 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779)
I20260929 11:20:34.815603 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.798569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53509 (local address 127.0.97.1:41211)
0929 11:20:34.808877 (+ 10308us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.808881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.808895 (+ 14us) server_negotiation.cc:407] Connection header received
0929 11:20:34.808942 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.808945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.808994 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.809070 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:34.809971 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.811031 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.813070 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.814596 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.814717 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.814827 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.815167 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.815300 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10195,"thread_start_us":10144,"threads_started":1}
I20260929 11:20:34.815770 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.798078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41211 (local address 127.0.97.2:53509)
0929 11:20:34.808324 (+ 10246us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.808344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.808459 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.809153 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.809158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.809203 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:34.809776 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.809788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.811151 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.811161 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.812923 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.812934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.814785 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.814792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.814981 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.815020 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.815132 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.815292 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10100,"mutex_wait_us":47,"thread_start_us":9676,"threads_started":1}
I20260929 11:20:34.816432 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50cecaa874e4b48a3ce224e7f4934c3" candidate_uuid: "9e882785921a49c48fb729bbbd15dae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613759af2a6042fcbb3da8c182311081" is_pre_election: true
I20260929 11:20:34.816712 1667 raft_consensus.cc:2454] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e882785921a49c48fb729bbbd15dae8 in term 0.
I20260929 11:20:34.816749 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.798148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53229 (local address 127.0.97.3:46779)
0929 11:20:34.798388 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.798397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.808191 (+ 9794us) server_negotiation.cc:407] Connection header received
0929 11:20:34.808411 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.808416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.808488 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.808577 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:34.809816 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.810925 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.813682 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.815329 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.815447 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.815536 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.816343 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.816474 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:20:34.816810 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.798015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46779 (local address 127.0.97.2:53229)
0929 11:20:34.808061 (+ 10046us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.808092 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.808263 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.809228 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.809231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.809249 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:34.809668 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.809678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.811107 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.811113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.813452 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.813467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.816034 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.816045 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.816182 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.816204 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.816300 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.816435 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9881,"mutex_wait_us":51,"thread_start_us":10357,"threads_started":1}
I20260929 11:20:34.817210 1738 leader_election.cc:302] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613759af2a6042fcbb3da8c182311081, 9e882785921a49c48fb729bbbd15dae8; no voters:
I20260929 11:20:34.817341 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50cecaa874e4b48a3ce224e7f4934c3" candidate_uuid: "9e882785921a49c48fb729bbbd15dae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d5e450300d4f9da698fd859d8d356b" is_pre_election: true
I20260929 11:20:34.817595 1950 raft_consensus.cc:2454] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e882785921a49c48fb729bbbd15dae8 in term 0.
I20260929 11:20:34.817637 2020 raft_consensus.cc:2781] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:34.817739 2020 raft_consensus.cc:482] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:34.817848 2020 raft_consensus.cc:3037] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.820009 2020 raft_consensus.cc:504] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.820604 2020 leader_election.cc:288] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [CANDIDATE]: Term 1 election: Requested vote from peers 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211), 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779)
I20260929 11:20:34.821033 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50cecaa874e4b48a3ce224e7f4934c3" candidate_uuid: "9e882785921a49c48fb729bbbd15dae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d5e450300d4f9da698fd859d8d356b"
I20260929 11:20:34.821094 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50cecaa874e4b48a3ce224e7f4934c3" candidate_uuid: "9e882785921a49c48fb729bbbd15dae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613759af2a6042fcbb3da8c182311081"
I20260929 11:20:34.821259 1667 raft_consensus.cc:3037] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.821259 1950 raft_consensus.cc:3037] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:34.824150 1667 raft_consensus.cc:2454] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e882785921a49c48fb729bbbd15dae8 in term 1.
I20260929 11:20:34.824164 1950 raft_consensus.cc:2454] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e882785921a49c48fb729bbbd15dae8 in term 1.
I20260929 11:20:34.824515 1738 leader_election.cc:302] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613759af2a6042fcbb3da8c182311081, 9e882785921a49c48fb729bbbd15dae8; no voters:
I20260929 11:20:34.824815 2020 raft_consensus.cc:2781] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:34.825255 2020 raft_consensus.cc:686] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [term 1 LEADER]: Becoming Leader. State: Replica: 9e882785921a49c48fb729bbbd15dae8, State: Running, Role: LEADER
I20260929 11:20:34.825665 2020 consensus_queue.cc:237] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } }
I20260929 11:20:34.827877 2060 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: TxnStatusTablet state changed. Reason: New leader 9e882785921a49c48fb729bbbd15dae8. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.828202 2060 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:20:34.828962 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:20:34.829161 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:20:34.829911 1510 catalog_manager.cc:5495] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } health_report { overall_health: UNKNOWN } } }
I20260929 11:20:34.857149 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.851356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.0.97.2:43093)
0929 11:20:34.851508 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.851512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.851761 (+ 249us) server_negotiation.cc:407] Connection header received
0929 11:20:34.852049 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.852053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.852154 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.852295 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:34.852661 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.853439 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.854675 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.855354 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.855495 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.855942 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.856456 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.856637 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.856736 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.856819 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:20:34.857136 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.851400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43093 (local address 127.0.0.1:41012)
0929 11:20:34.851671 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.851686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.851845 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.852313 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.852318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.852330 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:34.852495 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.852503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.853564 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.853572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.854542 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.854552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.855506 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.855511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.855675 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.855724 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.856566 (+ 842us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.856616 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.856706 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.856857 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 11:20:34.861783 1667 raft_consensus.cc:1261] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Refusing update from remote peer 9e882785921a49c48fb729bbbd15dae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:34.862018 1950 raft_consensus.cc:1261] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Refusing update from remote peer 9e882785921a49c48fb729bbbd15dae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:34.862641 2060 consensus_queue.cc:1035] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:34.863045 2020 consensus_queue.cc:1035] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:34.866496 2021 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: TxnStatusTablet state changed. Reason: New leader 9e882785921a49c48fb729bbbd15dae8. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.866772 2021 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.869541 2065 mvcc.cc:205] Tried to move back new op lower bound from 7334628699587067904 to 7334628699451318272. Current Snapshot: MvccSnapshot[applied={T|T < 7334628699587067904}]
I20260929 11:20:34.870956 2021 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.871212 2021 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 613759af2a6042fcbb3da8c182311081: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.876884 2034 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: TxnStatusTablet state changed. Reason: New leader 9e882785921a49c48fb729bbbd15dae8. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.877249 2034 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.879822 2072 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.880230 2072 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:20:34.880858 2060 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.881089 2060 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 9e882785921a49c48fb729bbbd15dae8: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:20:34.882611 2034 tablet_replica.cc:439] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e882785921a49c48fb729bbbd15dae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613759af2a6042fcbb3da8c182311081" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41211 } } peers { permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 } } peers { permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 } } }
I20260929 11:20:34.882834 2034 tablet_replica.cc:442] T e50cecaa874e4b48a3ce224e7f4934c3 P 50d5e450300d4f9da698fd859d8d356b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:34.894555 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.888430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.0.97.1:41211)
0929 11:20:34.888654 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.888661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.889103 (+ 442us) server_negotiation.cc:407] Connection header received
0929 11:20:34.889350 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.889354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.889425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.889568 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:34.889995 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.890727 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.892087 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.892506 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.892660 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.893079 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.893573 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.893729 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.893784 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.893858 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:20:34.910583 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.904228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52997 (local address 127.0.97.2:43093)
0929 11:20:34.904431 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.904440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.904976 (+ 536us) server_negotiation.cc:407] Connection header received
0929 11:20:34.905129 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.905133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.905190 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.905263 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:34.906225 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.906967 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.908964 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.909447 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.909534 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.910024 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.910108 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.910168 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.910317 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.910381 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:20:34.910815 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.904224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43093 (local address 127.0.97.1:52997)
0929 11:20:34.904872 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.904895 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.905022 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.905301 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.905306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.905318 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:34.905637 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.905647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.907091 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.907096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.908832 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.908847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.909615 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.909622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.909794 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.909866 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.910200 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.910211 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.910296 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.910429 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":390,"threads_started":1}
I20260929 11:20:34.919770 1950 raft_consensus.cc:1261] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Refusing update from remote peer 613759af2a6042fcbb3da8c182311081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:34.920109 1811 raft_consensus.cc:1261] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Refusing update from remote peer 613759af2a6042fcbb3da8c182311081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:34.920372 2021 consensus_queue.cc:1035] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d5e450300d4f9da698fd859d8d356b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:34.920696 2045 consensus_queue.cc:1035] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e882785921a49c48fb729bbbd15dae8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:34.958453 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.951445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37783 (local address 127.0.97.1:41211)
0929 11:20:34.951588 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:34.951592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:34.952278 (+ 686us) server_negotiation.cc:407] Connection header received
0929 11:20:34.952478 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:34.952483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:34.952553 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:34.952674 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:34.953456 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.954736 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.956437 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.957050 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.957147 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.957720 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.957828 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.957941 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:34.958075 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 11:20:34.958146 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 11:20:34.958591 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:34.951494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41211 (local address 127.0.97.2:37783)
0929 11:20:34.952187 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:34.952203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:34.952354 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:34.952800 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:34.952804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:34.952815 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:34.953220 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.953231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.954866 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.954871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.956260 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:34.956271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:34.957334 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:34.957339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:34.957491 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:34.957540 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:34.957941 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:34.957953 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 11:20:34.958055 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:20:34.958176 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":313,"threads_started":1}
I20260929 11:20:34.991676 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:34.991950 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:34.995350 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c379512bb874c62a6c43b7b5e73ad87: Sending DeleteTablet for 3 replicas of tablet b54c0d8b47ee46a7878c37231525fe51
I20260929 11:20:34.996333 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet b54c0d8b47ee46a7878c37231525fe51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58594
I20260929 11:20:34.996426 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet b54c0d8b47ee46a7878c37231525fe51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59006
I20260929 11:20:34.997027 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet b54c0d8b47ee46a7878c37231525fe51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41002
I20260929 11:20:35.011633 2087 tablet_replica.cc:330] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: stopping tablet replica
I20260929 11:20:35.011875 2087 raft_consensus.cc:2229] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:35.012157 2087 raft_consensus.cc:2258] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:35.013387 2087 ts_tablet_manager.cc:1905] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:35.020994 2087 ts_tablet_manager.cc:1918] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:35.021135 2087 log.cc:1196] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/b54c0d8b47ee46a7878c37231525fe51
I20260929 11:20:35.021582 2087 ts_tablet_manager.cc:1939] T b54c0d8b47ee46a7878c37231525fe51 P 50d5e450300d4f9da698fd859d8d356b: Deleting consensus metadata
I20260929 11:20:35.022696 1497 catalog_manager.cc:4841] TS 50d5e450300d4f9da698fd859d8d356b (127.0.97.3:46779): tablet b54c0d8b47ee46a7878c37231525fe51 (table test-table [id=a6c068c9315d4447b1e6b66285669b90]) successfully deleted
I20260929 11:20:35.025410 2089 tablet_replica.cc:330] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: stopping tablet replica
I20260929 11:20:35.025624 2089 raft_consensus.cc:2229] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:35.025934 2089 raft_consensus.cc:2258] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:35.028084 2089 ts_tablet_manager.cc:1905] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:35.030151 2088 tablet_replica.cc:330] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: stopping tablet replica
I20260929 11:20:35.030418 2088 raft_consensus.cc:2229] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:20:35.031020 2088 raft_consensus.cc:2258] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:35.032266 2088 ts_tablet_manager.cc:1905] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:35.037768 2089 ts_tablet_manager.cc:1918] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:35.037956 2089 log.cc:1196] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/b54c0d8b47ee46a7878c37231525fe51
I20260929 11:20:35.038450 2089 ts_tablet_manager.cc:1939] T b54c0d8b47ee46a7878c37231525fe51 P 9e882785921a49c48fb729bbbd15dae8: Deleting consensus metadata
I20260929 11:20:35.039623 1495 catalog_manager.cc:4841] TS 9e882785921a49c48fb729bbbd15dae8 (127.0.97.2:43093): tablet b54c0d8b47ee46a7878c37231525fe51 (table test-table [id=a6c068c9315d4447b1e6b66285669b90]) successfully deleted
I20260929 11:20:35.040354 2088 ts_tablet_manager.cc:1918] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:35.040493 2088 log.cc:1196] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/b54c0d8b47ee46a7878c37231525fe51
I20260929 11:20:35.041030 2088 ts_tablet_manager.cc:1939] T b54c0d8b47ee46a7878c37231525fe51 P 613759af2a6042fcbb3da8c182311081: Deleting consensus metadata
I20260929 11:20:35.042136 1495 catalog_manager.cc:4841] TS 613759af2a6042fcbb3da8c182311081 (127.0.97.1:41211): tablet b54c0d8b47ee46a7878c37231525fe51 (table test-table [id=a6c068c9315d4447b1e6b66285669b90]) successfully deleted
Sep 29 11:20:35 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:35.062614 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:35.026580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.0.97.1:41211)
0929 11:20:35.026769 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:35.026786 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:35.043463 (+ 16677us) server_negotiation.cc:407] Connection header received
0929 11:20:35.043737 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:35.043747 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:35.043844 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:35.043960 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:35.045850 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.047014 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.048960 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.049625 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.049755 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:35.054809 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:35.054824 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:35.054846 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:35.054891 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:35.059266 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.060234 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.060244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.060248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.060415 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.060882 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.060887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.060891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.061373 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:35.061670 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:35.062013 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 11:20:35.062232 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 11:20:35.063948 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59034
I20260929 11:20:35.086843 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:35.068045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.97.62:45715)
0929 11:20:35.068226 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:35.068231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:35.070290 (+ 2059us) server_negotiation.cc:407] Connection header received
0929 11:20:35.070548 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:35.070554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:35.070650 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:35.070755 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:35.072496 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.073664 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.075946 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.076643 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.076772 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:35.079897 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:35.079904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:35.079909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:35.079948 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:35.083428 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.084710 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.084717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.084722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.084866 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.085308 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.085316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.085319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.085657 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:35.085890 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:35.086287 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 11:20:35.086495 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 11:20:35.087402 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57404
I20260929 11:20:35.089702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1581
I20260929 11:20:35.109411 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:35.091515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.0.97.1:41211)
0929 11:20:35.091683 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:35.091689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:35.093461 (+ 1772us) server_negotiation.cc:407] Connection header received
0929 11:20:35.093616 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:35.093623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:35.093745 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:35.093865 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:35.095303 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.096489 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.098301 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.099046 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.099187 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:35.102076 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:35.102084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:35.102089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:35.102132 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:35.105990 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.106768 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.106774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.106778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.106948 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.107385 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.107390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.107403 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.107795 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:35.108512 (+ 717us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:35.108731 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 11:20:35.108957 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 11:20:35.110373 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59048
W20260929 11:20:35.111027 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1581: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:35.111222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1581
I20260929 11:20:35.130714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1721
Sep 29 11:20:35 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:20:35.147565 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:35.131549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.0.97.2:43093)
0929 11:20:35.131776 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:35.131782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:35.131830 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:20:35.131928 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:35.131938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:35.132029 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:35.132136 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:35.133365 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.134527 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.135806 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.136502 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.136676 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:35.140684 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:35.140697 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:35.140715 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:35.140767 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:35.144806 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.145404 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.145413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.145417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.145627 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.145970 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.145975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.145979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.146424 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:35.146746 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:35.146912 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 11:20:35.147152 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 11:20:35.148397 1851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41028
W20260929 11:20:35.149134 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:35.149319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1721
I20260929 11:20:35.169576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1860
Sep 29 11:20:35 dist-test-slave-srst krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:20:35.183914 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:35.170396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.0.97.3:46779)
0929 11:20:35.170576 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:35.170582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:35.170603 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:20:35.170702 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:35.170707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:35.170788 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:35.170907 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:35.172126 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.172948 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.174145 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:35.174592 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:35.174724 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:35.178654 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:35.178666 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:35.178684 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:35.178713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:35.181343 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.181975 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.181988 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.181990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.182098 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:35.182459 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:35.182466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:35.182469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:35.182792 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:35.183107 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:35.183375 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 11:20:35.183587 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 11:20:35.184752 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604
W20260929 11:20:35.185412 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:35.185564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1860
I20260929 11:20:35.213039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 1478
I20260929 11:20:35.214131 1550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:35.327699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 1478
2026-09-29T11:20:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4785 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:35 dist-test-slave-srst krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:35 dist-test-slave-srst krb5kdc[2098](info): set up 2 sockets
Sep 29 11:20:35 dist-test-slave-srst krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260929 11:20:36.408788 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:36 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:36.583585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41101
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:36.890261 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:36.890532 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:36.890614 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:36.897436 2111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:36.897528 2111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:20:36.897569 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:36.897629 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:36.897675 2111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:36.897728 2111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:36.906731 2111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41101
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:36.907847 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:36.909065 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:36.920382 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:36.920444 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:36.920466 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:36.921253 2111 server_base.cc:1034] running on GCE node
I20260929 11:20:36.921990 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:36.923311 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:36.924563 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790680836924486 us; error 59 us; skew 500 ppm
I20260929 11:20:36.924916 2111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:36 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:36.927980 2111 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:36.929366 2111 webserver.cc:460] Webserver started at http://127.0.97.62:43309/ using document root <none> and password file <none>
I20260929 11:20:36.930053 2111 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:36.930179 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:36.930480 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:36.933312 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "b932712a13224553a7ee3eaab22ea73e"
format_stamp: "Formatted at 2026-09-29 11:20:36 on dist-test-slave-srst"
server_key: "71ba3a75cb24444da397cc5064f153f5"
server_key_iv: "2062c14ae70132ef52922425d34a829a"
server_key_version: "encryptionkey@0"
I20260929 11:20:36.934047 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "b932712a13224553a7ee3eaab22ea73e"
format_stamp: "Formatted at 2026-09-29 11:20:36 on dist-test-slave-srst"
server_key: "71ba3a75cb24444da397cc5064f153f5"
server_key_iv: "2062c14ae70132ef52922425d34a829a"
server_key_version: "encryptionkey@0"
I20260929 11:20:36.938907 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 11:20:36.942126 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:36.943357 2111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:20:36.943552 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "b932712a13224553a7ee3eaab22ea73e"
format_stamp: "Formatted at 2026-09-29 11:20:36 on dist-test-slave-srst"
server_key: "71ba3a75cb24444da397cc5064f153f5"
server_key_iv: "2062c14ae70132ef52922425d34a829a"
server_key_version: "encryptionkey@0"
I20260929 11:20:36.943770 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:36.974099 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:36.978518 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:36.978787 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:36.999503 2111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41101
I20260929 11:20:36.999523 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41101 every 8 connection(s)
I20260929 11:20:37.000912 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:37.004475 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:37.008222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 2111
I20260929 11:20:37.008420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:37.009229 388 external_mini_cluster.cc:1428] Setting key 5b90105fe10e6e6789bde67a4edb79df
I20260929 11:20:37.017952 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Bootstrap starting.
I20260929 11:20:37.021718 2188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:37.023047 2188 log.cc:824] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:20:37 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:37.026103 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: No bootstrap required, opened a new log
I20260929 11:20:37.031915 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:37.011426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.0.97.62:41101)
0929 11:20:37.012217 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:37.012229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:37.012270 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:20:37.013880 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:37.013902 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:37.014308 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:37.014641 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:37.016339 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.017680 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.019359 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.020055 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.020181 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:37.024983 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:37.025000 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:37.025013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:37.025055 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:37.028101 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:37.028912 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:37.028918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:37.028924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:37.029097 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:37.029451 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:37.029454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:37.029456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:37.029941 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:37.030159 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:37.030559 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 11:20:37.030912 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":375,"threads_started":1}
I20260929 11:20:37.034633 2188 raft_consensus.cc:348] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b932712a13224553a7ee3eaab22ea73e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41101 } }
I20260929 11:20:37.034979 2188 raft_consensus.cc:374] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:37.035084 2188 raft_consensus.cc:729] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b932712a13224553a7ee3eaab22ea73e, State: Initialized, Role: FOLLOWER
I20260929 11:20:37.035616 2188 consensus_queue.cc:260] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b932712a13224553a7ee3eaab22ea73e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41101 } }
I20260929 11:20:37.035856 2188 raft_consensus.cc:388] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:37.035990 2188 raft_consensus.cc:482] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:37.036164 2188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:37.038215 2188 raft_consensus.cc:504] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b932712a13224553a7ee3eaab22ea73e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41101 } }
I20260929 11:20:37.038686 2188 leader_election.cc:302] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b932712a13224553a7ee3eaab22ea73e; no voters:
I20260929 11:20:37.039261 2188 leader_election.cc:288] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:37.039390 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:37.040334 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [term 1 LEADER]: Becoming Leader. State: Replica: b932712a13224553a7ee3eaab22ea73e, State: Running, Role: LEADER
I20260929 11:20:37.040835 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b932712a13224553a7ee3eaab22ea73e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41101 } }
I20260929 11:20:37.041317 2188 sys_catalog.cc:564] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:37.043540 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b932712a13224553a7ee3eaab22ea73e. Latest consensus state: current_term: 1 leader_uuid: "b932712a13224553a7ee3eaab22ea73e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b932712a13224553a7ee3eaab22ea73e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41101 } } }
I20260929 11:20:37.043520 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b932712a13224553a7ee3eaab22ea73e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b932712a13224553a7ee3eaab22ea73e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41101 } } }
I20260929 11:20:37.043955 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:37.043955 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:37.049523 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:37.055087 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:37.063007 2200 catalog_manager.cc:1347] Generated new cluster ID: 53b5abfe76f143fe995622690ff44877
I20260929 11:20:37.063123 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:37.075788 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:37.077204 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:37.083089 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Generated new TSK 0
I20260929 11:20:37.083995 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:37.126120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41101
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:37.447134 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:37.447422 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:37.447526 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:37.454931 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:37.455040 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:37.455168 2214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:20:37.455243 2214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:20:37.455348 2214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:37.464787 2214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:37.466051 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:37.467378 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:37.479414 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:37.479461 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:37.479503 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:37.480310 2214 server_base.cc:1034] running on GCE node
I20260929 11:20:37.480959 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:37.481885 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:37.483091 2214 hybrid_clock.cc:648] HybridClock initialized: now 1790680837483028 us; error 60 us; skew 500 ppm
I20260929 11:20:37.483377 2214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:37 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:37.486420 2214 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:37.487679 2214 webserver.cc:460] Webserver started at http://127.0.97.1:36309/ using document root <none> and password file <none>
I20260929 11:20:37.488282 2214 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:37.488376 2214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:37.488675 2214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:37.491317 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "67cf37ace4b34aed977f816b712e9559"
format_stamp: "Formatted at 2026-09-29 11:20:37 on dist-test-slave-srst"
server_key: "c202b13b680fe6c202db099eedd29bd3"
server_key_iv: "87cf8d97ebe86b964e0da517b76300c9"
server_key_version: "encryptionkey@0"
I20260929 11:20:37.491986 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "67cf37ace4b34aed977f816b712e9559"
format_stamp: "Formatted at 2026-09-29 11:20:37 on dist-test-slave-srst"
server_key: "c202b13b680fe6c202db099eedd29bd3"
server_key_iv: "87cf8d97ebe86b964e0da517b76300c9"
server_key_version: "encryptionkey@0"
I20260929 11:20:37.496680 2214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 11:20:37.499712 2229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:37.500929 2214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 11:20:37.501075 2214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "67cf37ace4b34aed977f816b712e9559"
format_stamp: "Formatted at 2026-09-29 11:20:37 on dist-test-slave-srst"
server_key: "c202b13b680fe6c202db099eedd29bd3"
server_key_iv: "87cf8d97ebe86b964e0da517b76300c9"
server_key_version: "encryptionkey@0"
I20260929 11:20:37.501246 2214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:37.519630 2214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:37.523816 2214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:37.524084 2214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:37.526513 2214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:37.526639 2214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:37.526757 2214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:37.526835 2214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:20:37 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:37.552459 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:37.527301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52089 (local address 127.0.97.62:41101)
0929 11:20:37.527504 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:37.527509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:37.528945 (+ 1436us) server_negotiation.cc:407] Connection header received
0929 11:20:37.531002 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:37.531008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:37.531083 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:37.531192 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:37.533911 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.535163 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.538508 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.539192 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.539343 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:37.545170 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:37.545177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:37.545182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:37.545226 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:37.548880 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:37.549968 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:37.549974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:37.549978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:37.550173 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:37.550684 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:37.550689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:37.550693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:37.550961 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:37.551138 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:37.551858 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 11:20:37.552090 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:20:37.553573 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:37.527904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.97.1:52089)
0929 11:20:37.528799 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:37.528839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:37.530690 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:37.531662 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:37.531679 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:37.532586 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:37.533645 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:37.533689 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.535399 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.535409 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:37.538256 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:37.538272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.539653 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.539663 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:37.540074 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:37.541063 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:37.541105 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:37.544843 (+ 3738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:37.549104 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:37.549111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:37.549123 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:37.549763 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:37.550316 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:37.550321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:37.550325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:37.550561 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:37.551114 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:37.551123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:37.551491 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 11:20:37.551764 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 11:20:37.552325 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":389,"threads_started":1}
I20260929 11:20:37.573661 2214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44903
I20260929 11:20:37.573683 2348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44903 every 8 connection(s)
I20260929 11:20:37.574970 2214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:37.577066 2349 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41101
I20260929 11:20:37.577287 2349 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:37.578119 2349 heartbeater.cc:511] Master 127.0.97.62:41101 requested a full tablet report, sending...
I20260929 11:20:37.579366 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:37.559422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53349 (local address 127.0.97.62:41101)
0929 11:20:37.559563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:37.559567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:37.565116 (+ 5549us) server_negotiation.cc:407] Connection header received
0929 11:20:37.565334 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:37.565341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:37.565405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:37.565503 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:37.566810 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.568068 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.570849 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.571486 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.571620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:37.573745 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:37.573751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:37.573756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:37.573795 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:37.576443 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:37.577281 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:37.577284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:37.577286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:37.577469 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:37.577900 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:37.577904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:37.577906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:37.578160 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:37.578319 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:37.578808 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 11:20:37.579058 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 11:20:37.579401 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:37.559461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.97.1:53349)
0929 11:20:37.564997 (+ 5536us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:37.565012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:37.565197 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:37.565528 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:37.565533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:37.566213 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:37.566646 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:37.566662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.568257 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.568268 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:37.570577 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:37.570597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:37.571700 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:37.571709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:37.571897 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:37.572654 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:37.572682 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:37.573562 (+ 880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:37.576625 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:37.576630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:37.576634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:37.577123 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:37.577571 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:37.577576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:37.577579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:37.577777 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:37.578345 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:37.578350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:37.578554 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 11:20:37.578706 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 11:20:37.578992 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5416,"thread_start_us":5280,"threads_started":1}
I20260929 11:20:37.580314 2143 ts_manager.cc:194] Registered new tserver with Master: 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903)
I20260929 11:20:37.581892 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52089
I20260929 11:20:37.583178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 2214
I20260929 11:20:37.583395 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:37.583968 388 external_mini_cluster.cc:1428] Setting key e8289b114225cce828f123b4c7f8b1f9
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:37.621764 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41101
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:37.929320 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:37.929571 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:37.929744 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:37.936864 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:37.936980 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:37.937137 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:20:37.937224 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:20:37.937353 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:37.947091 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:37.948503 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:37.950002 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:37.962730 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:37.962761 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:37.963204 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:37.963624 2353 server_base.cc:1034] running on GCE node
I20260929 11:20:37.964141 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:37.965015 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:37.966199 2353 hybrid_clock.cc:648] HybridClock initialized: now 1790680837966141 us; error 58 us; skew 500 ppm
I20260929 11:20:37.966456 2353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:37 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:37.969395 2353 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:37.970829 2353 webserver.cc:460] Webserver started at http://127.0.97.2:32859/ using document root <none> and password file <none>
I20260929 11:20:37.971465 2353 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:37.971583 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:37.971903 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:37.974668 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ca2e35d549f4cb1abae316482fd9743"
format_stamp: "Formatted at 2026-09-29 11:20:37 on dist-test-slave-srst"
server_key: "aabc7bce01177b4c8437794174c4aade"
server_key_iv: "977547bfb2b84ad162092d8c991a8ad4"
server_key_version: "encryptionkey@0"
I20260929 11:20:37.975466 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ca2e35d549f4cb1abae316482fd9743"
format_stamp: "Formatted at 2026-09-29 11:20:37 on dist-test-slave-srst"
server_key: "aabc7bce01177b4c8437794174c4aade"
server_key_iv: "977547bfb2b84ad162092d8c991a8ad4"
server_key_version: "encryptionkey@0"
I20260929 11:20:37.980295 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 11:20:37.983570 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:37.984833 2353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:20:37.984979 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "8ca2e35d549f4cb1abae316482fd9743"
format_stamp: "Formatted at 2026-09-29 11:20:37 on dist-test-slave-srst"
server_key: "aabc7bce01177b4c8437794174c4aade"
server_key_iv: "977547bfb2b84ad162092d8c991a8ad4"
server_key_version: "encryptionkey@0"
I20260929 11:20:37.985162 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:38.007897 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:38.013510 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:38.013787 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:38.016757 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:38.016875 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:38.016976 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:38.017064 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:20:38 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:38.039691 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.016997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48423 (local address 127.0.97.62:41101)
0929 11:20:38.017181 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.017187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.018594 (+ 1407us) server_negotiation.cc:407] Connection header received
0929 11:20:38.020669 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.020677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.020760 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.020841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:38.023067 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.023878 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.026487 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.026917 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.027052 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.032376 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:38.032380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:38.032384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:38.032412 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:38.036096 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.037017 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.037022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.037024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.037175 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.037711 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.037717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.037720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.038122 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:38.038267 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.039146 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.039368 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:20:38.040753 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.017648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.97.2:48423)
0929 11:20:38.018439 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.018489 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.020325 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.021319 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.021329 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.022047 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:38.022893 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.022926 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.024032 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.024038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.026349 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.026362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.027135 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.027142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.027450 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.028295 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:38.028332 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:38.032124 (+ 3792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:38.036245 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.036251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.036262 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.036834 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.037332 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.037342 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.037345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.037594 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.038241 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:38.038248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:38.038759 (+ 511us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.039054 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.039560 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":342,"threads_started":1}
I20260929 11:20:38.065012 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.051252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.97.2:42539)
0929 11:20:38.051873 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.051892 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.052010 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.052377 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.052380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.052848 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:38.053377 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.053387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.054601 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.054605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.055546 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.055556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.056423 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.056428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.056581 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.057357 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:38.057394 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:38.058749 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:38.062012 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.062017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.062022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.062666 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.063050 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.063055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.063059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.063193 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.063716 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:38.063721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:38.063936 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.064163 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.064481 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":347,"threads_started":1}
I20260929 11:20:38.065078 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.051242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42539 (local address 127.0.97.62:41101)
0929 11:20:38.051463 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.051471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.051980 (+ 509us) server_negotiation.cc:407] Connection header received
0929 11:20:38.052122 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.052131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.052225 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.052357 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:38.053507 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.054474 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.055723 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.056243 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.056381 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.058931 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:38.058935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:38.058938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:38.058966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:38.061902 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.062798 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.062806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.062819 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.062942 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.063300 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.063307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.063311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.063603 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:38.063745 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.064309 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.064637 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:20:38.074227 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37041
I20260929 11:20:38.074260 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37041 every 8 connection(s)
I20260929 11:20:38.075439 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:38.077391 2488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41101
I20260929 11:20:38.077608 2488 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:38.077939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 2353
I20260929 11:20:38.078089 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:38.078302 2488 heartbeater.cc:511] Master 127.0.97.62:41101 requested a full tablet report, sending...
I20260929 11:20:38.078568 388 external_mini_cluster.cc:1428] Setting key 809651e42b3d5166ae1d536b5eee80f4
I20260929 11:20:38.079639 2143 ts_manager.cc:194] Registered new tserver with Master: 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041)
I20260929 11:20:38.080476 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48423
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:38.115712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41101
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:38.422366 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:38.422611 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:38.422770 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:38.429598 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:38.429693 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:38.429865 2492 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:20:38.429972 2492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:20:38.430102 2492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:38.439291 2492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:38.440536 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:38.441957 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:38.454774 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:38.454841 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:38.454869 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:38.455628 2492 server_base.cc:1034] running on GCE node
I20260929 11:20:38.456225 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:38.457203 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:38.458384 2492 hybrid_clock.cc:648] HybridClock initialized: now 1790680838458328 us; error 49 us; skew 500 ppm
I20260929 11:20:38.458663 2492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:38 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:38.466851 2492 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:38.468170 2492 webserver.cc:460] Webserver started at http://127.0.97.3:38661/ using document root <none> and password file <none>
I20260929 11:20:38.468870 2492 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:38.468966 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:38.469213 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:38.471910 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "9807cabe70404541b44412b7a58432e6"
format_stamp: "Formatted at 2026-09-29 11:20:38 on dist-test-slave-srst"
server_key: "0af21916efb3e3e35f312bb647053b80"
server_key_iv: "60a20cd72a276108213271da6ce2c555"
server_key_version: "encryptionkey@0"
I20260929 11:20:38.472795 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9807cabe70404541b44412b7a58432e6"
format_stamp: "Formatted at 2026-09-29 11:20:38 on dist-test-slave-srst"
server_key: "0af21916efb3e3e35f312bb647053b80"
server_key_iv: "60a20cd72a276108213271da6ce2c555"
server_key_version: "encryptionkey@0"
I20260929 11:20:38.477495 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:20:38.480757 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:38.482075 2492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:20:38.482250 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "9807cabe70404541b44412b7a58432e6"
format_stamp: "Formatted at 2026-09-29 11:20:38 on dist-test-slave-srst"
server_key: "0af21916efb3e3e35f312bb647053b80"
server_key_iv: "60a20cd72a276108213271da6ce2c555"
server_key_version: "encryptionkey@0"
I20260929 11:20:38.482448 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:38.505347 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:38.509428 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:38.509714 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:38.512267 2492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:38.512387 2492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:38.512516 2492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:38.512612 2492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:20:38 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:38.536038 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.512709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53419 (local address 127.0.97.62:41101)
0929 11:20:38.512916 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.512924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.514313 (+ 1389us) server_negotiation.cc:407] Connection header received
0929 11:20:38.516404 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.516409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.516493 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.516596 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:38.518718 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.520005 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.522560 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.523376 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.523520 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.529083 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:38.529088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:38.529091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:38.529130 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:38.532127 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.532910 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.532915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.532919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.533101 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.533693 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.533699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.533704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.534051 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:38.534208 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.535229 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.535540 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:20:38.536803 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.513352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.97.3:53419)
0929 11:20:38.514156 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.514193 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.516067 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.516857 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.516888 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.517635 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:38.518519 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.518547 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.520182 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.520189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.522442 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.522457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.523889 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.523899 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.524275 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.525166 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:38.525232 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:38.528834 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:38.532256 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.532266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.532282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.532774 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.533243 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.533251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.533256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.533535 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.534190 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:38.534197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:38.534696 (+ 499us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.535048 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.535502 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":376,"threads_started":1}
I20260929 11:20:38.556480 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.544091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58719 (local address 127.0.97.62:41101)
0929 11:20:38.544346 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.544354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.545000 (+ 646us) server_negotiation.cc:407] Connection header received
0929 11:20:38.545206 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.545215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.545271 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.545368 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:38.546250 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.546933 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.548354 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.548856 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.548987 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.551113 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:38.551120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:38.551124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:38.551162 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:38.553725 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.554338 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.554346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.554351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.554499 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.554842 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.554849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.554852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.555165 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:38.555326 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.555822 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.556030 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:20:38.557570 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.544155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.97.3:58719)
0929 11:20:38.544886 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.544915 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.545077 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.545388 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.545392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.545843 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:38.546099 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.546107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.547075 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.547080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.548182 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.548194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.549052 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.549059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.549223 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.549916 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:38.549933 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:38.550934 (+ 1001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:38.553849 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.553856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.553859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.554224 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.554604 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.554610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.554613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.554741 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.555280 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:38.555285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:38.555492 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.556372 (+ 880us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.556929 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":363,"threads_started":1}
I20260929 11:20:38.572095 2492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34339
I20260929 11:20:38.572130 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34339 every 8 connection(s)
I20260929 11:20:38.573383 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:38.575430 2627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41101
I20260929 11:20:38.575639 2627 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:38.576223 2627 heartbeater.cc:511] Master 127.0.97.62:41101 requested a full tablet report, sending...
I20260929 11:20:38.577557 2143 ts_manager.cc:194] Registered new tserver with Master: 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:38.578322 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53419
I20260929 11:20:38.581264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 2492
I20260929 11:20:38.581415 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:38.581879 388 external_mini_cluster.cc:1428] Setting key 20d8333cc599c9c9751b019c6d2f11aa
I20260929 11:20:38.584695 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:20:38.584909 2349 heartbeater.cc:503] Master 127.0.97.62:41101 was elected leader, sending a full tablet report...
Sep 29 11:20:38 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:20:38 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:38.628422 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.615454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.0.97.62:41101)
0929 11:20:38.615618 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.615622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.615963 (+ 341us) server_negotiation.cc:407] Connection header received
0929 11:20:38.616131 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.616134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.616209 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.616300 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:38.617725 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.618459 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.619693 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.620113 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.620203 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.623757 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:38.623762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:38.623765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:38.623804 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:38.626389 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.626960 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.626963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.626966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.627071 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.627370 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.627373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.627375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.627629 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:38.627817 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.628051 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.628197 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:20:38.632282 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:38.638839 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:38.664976 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.651420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34339 (local address 127.0.0.1:56446)
0929 11:20:38.652796 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.652822 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.652975 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.653960 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.653967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.653998 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:38.654475 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.654490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.657312 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.657320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.660530 (+ 3210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.660564 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.663607 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.663616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.663894 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.663965 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.664125 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.664383 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":1,"thread_start_us":1001,"threads_started":1}
I20260929 11:20:38.666749 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.650435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37041 (local address 127.0.0.1:53634)
0929 11:20:38.651336 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.651377 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.651577 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.652170 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.652177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.652204 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:38.652778 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.652800 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.657781 (+ 4981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.657789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.662079 (+ 4290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.662099 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.665698 (+ 3599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.665707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.665972 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.666023 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.666098 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.666279 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":524,"threads_started":1}
I20260929 11:20:38.666764 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.651862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.0.97.3:34339)
0929 11:20:38.653427 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.653435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.653464 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:20:38.653557 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.653563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.653759 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.653980 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:38.654734 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.657090 (+ 2356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.660846 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.662998 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.663153 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.663462 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.666075 (+ 2613us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.666315 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1397,"thread_start_us":1229,"threads_started":1}
I20260929 11:20:38.669435 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.650608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.0.97.2:37041)
0929 11:20:38.651220 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.651229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.651719 (+ 490us) server_negotiation.cc:407] Connection header received
0929 11:20:38.651794 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.651801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.651986 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.652165 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:38.652966 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.657587 (+ 4621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.663526 (+ 5939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.665391 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.667167 (+ 1776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.668732 (+ 1565us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.668826 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.669041 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":283,"threads_started":1}
I20260929 11:20:38.671456 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 3efdd8d34ced4c73a9793c471678395d (DEFAULT_TABLE table=test-table [id=d85b33c7a83444dda300fb7a1a63d330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:38.673286 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efdd8d34ced4c73a9793c471678395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:38.675034 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 3efdd8d34ced4c73a9793c471678395d (DEFAULT_TABLE table=test-table [id=d85b33c7a83444dda300fb7a1a63d330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:38.675693 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.653843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44903 (local address 127.0.0.1:46656)
0929 11:20:38.665743 (+ 11900us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.665766 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.665882 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.670492 (+ 4610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.670496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.670519 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:38.670704 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.670719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.671994 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.671998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.673283 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.673307 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.674991 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.674998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.675180 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.675209 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.675314 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.675439 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11776,"thread_start_us":16962,"threads_started":1}
I20260929 11:20:38.676270 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.654027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.0.97.1:44903)
0929 11:20:38.670022 (+ 15995us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.670029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.670062 (+ 33us) server_negotiation.cc:407] Connection header received
0929 11:20:38.670129 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.670151 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.670316 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.670512 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:38.670895 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.671839 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.673505 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.674828 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.674961 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.675160 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.675354 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.675500 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15855,"thread_start_us":15727,"threads_started":1}
I20260929 11:20:38.676297 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efdd8d34ced4c73a9793c471678395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:38.679395 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 3efdd8d34ced4c73a9793c471678395d (DEFAULT_TABLE table=test-table [id=d85b33c7a83444dda300fb7a1a63d330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:38.680622 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efdd8d34ced4c73a9793c471678395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:38.687014 2646 tablet_bootstrap.cc:492] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Bootstrap starting.
I20260929 11:20:38.687855 2647 tablet_bootstrap.cc:492] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Bootstrap starting.
I20260929 11:20:38.690567 2646 tablet_bootstrap.cc:654] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:38.691162 2647 tablet_bootstrap.cc:654] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:38.691797 2646 log.cc:824] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:38.692312 2647 log.cc:824] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:38.696291 2649 tablet_bootstrap.cc:492] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Bootstrap starting.
I20260929 11:20:38.699311 2649 tablet_bootstrap.cc:654] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:38.699852 2646 tablet_bootstrap.cc:492] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: No bootstrap required, opened a new log
I20260929 11:20:38.699862 2647 tablet_bootstrap.cc:492] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: No bootstrap required, opened a new log
I20260929 11:20:38.700111 2646 ts_tablet_manager.cc:1397] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260929 11:20:38.700111 2647 ts_tablet_manager.cc:1397] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260929 11:20:38.700340 2649 log.cc:824] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:38.702688 2649 tablet_bootstrap.cc:492] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: No bootstrap required, opened a new log
I20260929 11:20:38.702936 2649 ts_tablet_manager.cc:1397] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 11:20:38.710049 2649 raft_consensus.cc:348] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.710410 2649 raft_consensus.cc:374] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:38.710500 2649 raft_consensus.cc:729] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cf37ace4b34aed977f816b712e9559, State: Initialized, Role: FOLLOWER
I20260929 11:20:38.710588 2646 raft_consensus.cc:348] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.710588 2647 raft_consensus.cc:348] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.711006 2646 raft_consensus.cc:374] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:38.711006 2647 raft_consensus.cc:374] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:38.711105 2646 raft_consensus.cc:729] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Initialized, Role: FOLLOWER
I20260929 11:20:38.711105 2647 raft_consensus.cc:729] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9807cabe70404541b44412b7a58432e6, State: Initialized, Role: FOLLOWER
I20260929 11:20:38.711162 2649 consensus_queue.cc:260] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.711694 2646 consensus_queue.cc:260] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.711663 2647 consensus_queue.cc:260] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.712885 2649 ts_tablet_manager.cc:1428] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 11:20:38.716571 2488 heartbeater.cc:503] Master 127.0.97.62:41101 was elected leader, sending a full tablet report...
I20260929 11:20:38.717324 2627 heartbeater.cc:503] Master 127.0.97.62:41101 was elected leader, sending a full tablet report...
I20260929 11:20:38.717659 2647 ts_tablet_manager.cc:1428] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Time spent starting tablet: real 0.017s user 0.018s sys 0.001s
I20260929 11:20:38.719137 2646 ts_tablet_manager.cc:1428] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Time spent starting tablet: real 0.019s user 0.008s sys 0.010s
I20260929 11:20:38.782981 2653 raft_consensus.cc:482] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:38.783346 2653 raft_consensus.cc:504] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.784982 2653 leader_election.cc:288] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339), 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903)
I20260929 11:20:38.794097 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.785456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37751 (local address 127.0.97.3:34339)
0929 11:20:38.785678 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.785684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.786454 (+ 770us) server_negotiation.cc:407] Connection header received
0929 11:20:38.786716 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.786721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.786791 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.786886 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:38.787653 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.788770 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.791686 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.792957 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.793086 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.793189 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.793718 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.793853 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 11:20:38.794094 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.785451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53319 (local address 127.0.97.1:44903)
0929 11:20:38.785678 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.785684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.786561 (+ 877us) server_negotiation.cc:407] Connection header received
0929 11:20:38.786678 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.786684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.786776 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.786888 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:38.787653 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.788761 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.791686 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.792957 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.793086 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.793189 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.793718 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.793863 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 11:20:38.794404 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.785455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44903 (local address 127.0.97.2:53319)
0929 11:20:38.786444 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.786463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.786560 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.786906 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.786909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.786933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:38.787500 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.787512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.788884 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.788889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.791527 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.791542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.793222 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.793229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.793488 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.793580 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.793678 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.793934 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":248,"thread_start_us":455,"threads_started":1}
I20260929 11:20:38.794842 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.785425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34339 (local address 127.0.97.2:37751)
0929 11:20:38.786327 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.786347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.786518 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.786902 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.786907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.786998 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:38.787500 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.787512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.788886 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.788889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.791527 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.791542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.793232 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.793236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.793491 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.793583 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.793678 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.793934 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":334,"spinlock_wait_cycles":22784,"thread_start_us":503,"threads_started":1}
I20260929 11:20:38.795465 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efdd8d34ced4c73a9793c471678395d" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559" is_pre_election: true
I20260929 11:20:38.795763 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efdd8d34ced4c73a9793c471678395d" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6" is_pre_election: true
I20260929 11:20:38.796000 2299 raft_consensus.cc:2454] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 0.
I20260929 11:20:38.796254 2582 raft_consensus.cc:2454] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 0.
I20260929 11:20:38.796725 2369 leader_election.cc:302] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cf37ace4b34aed977f816b712e9559, 8ca2e35d549f4cb1abae316482fd9743; no voters:
I20260929 11:20:38.797150 2653 raft_consensus.cc:2781] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:38.797325 2653 raft_consensus.cc:482] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:38.797441 2653 raft_consensus.cc:3037] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:38.799554 2653 raft_consensus.cc:504] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.800107 2653 leader_election.cc:288] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 election: Requested vote from peers 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339), 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903)
I20260929 11:20:38.800539 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efdd8d34ced4c73a9793c471678395d" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6"
I20260929 11:20:38.800745 2582 raft_consensus.cc:3037] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:38.800876 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efdd8d34ced4c73a9793c471678395d" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559"
I20260929 11:20:38.801066 2299 raft_consensus.cc:3037] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:38.803859 2582 raft_consensus.cc:2454] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 1.
I20260929 11:20:38.804062 2299 raft_consensus.cc:2454] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 1.
I20260929 11:20:38.804245 2372 leader_election.cc:302] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca2e35d549f4cb1abae316482fd9743, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:38.804577 2653 raft_consensus.cc:2781] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:38.805075 2653 raft_consensus.cc:686] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Running, Role: LEADER
I20260929 11:20:38.805598 2653 consensus_queue.cc:237] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } }
I20260929 11:20:38.810159 2143 catalog_manager.cc:5495] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ca2e35d549f4cb1abae316482fd9743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } health_report { overall_health: UNKNOWN } } }
W20260929 11:20:38.824911 2628 tablet.cc:2369] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:20:38.827379 2350 tablet.cc:2369] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:20:38.827378 2489 tablet.cc:2369] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:20:38 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:38.893198 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.879174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.0.97.62:41101)
0929 11:20:38.879370 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:38.879374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:38.879868 (+ 494us) server_negotiation.cc:407] Connection header received
0929 11:20:38.880079 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:38.880084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:38.880149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:38.880261 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:38.881580 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.882353 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.883673 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.884104 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.884247 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.887972 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:38.887977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:38.887981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:38.888008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:38.890524 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.891170 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.891178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.891181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.891300 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:38.891674 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:38.891680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:38.891686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:38.891946 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:38.892091 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:38.892464 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 11:20:38.892833 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":73}
I20260929 11:20:38.893254 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:38.879172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41101 (local address 127.0.0.1:42518)
0929 11:20:38.879760 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:38.879776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:38.879951 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:38.880283 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:38.880287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:38.881136 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:38.881431 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.881442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.882512 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.882517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.883524 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:38.883535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:38.884280 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:38.884285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:38.884524 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:38.885303 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:38.885321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:38.887773 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:38.890655 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.890666 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.890674 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.891055 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.891400 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:38.891407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:38.891410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:38.891563 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:38.892065 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:38.892073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:38.892229 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 11:20:38.892389 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 11:20:38.892711 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":348,"threads_started":1}
I20260929 11:20:38.898367 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42518:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:20:38.899375 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:38.905619 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 87aa4e54c037460faeb6b23142372cc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72dd13e5394840c3b1a786c55911acad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:20:38.905411 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 87aa4e54c037460faeb6b23142372cc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72dd13e5394840c3b1a786c55911acad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:20:38.905910 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 87aa4e54c037460faeb6b23142372cc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72dd13e5394840c3b1a786c55911acad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:20:38.906471 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aa4e54c037460faeb6b23142372cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:38.906471 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aa4e54c037460faeb6b23142372cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:38.906775 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aa4e54c037460faeb6b23142372cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:38.912258 2647 tablet_bootstrap.cc:492] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: Bootstrap starting.
I20260929 11:20:38.913410 2646 tablet_bootstrap.cc:492] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: Bootstrap starting.
I20260929 11:20:38.913439 2649 tablet_bootstrap.cc:492] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: Bootstrap starting.
I20260929 11:20:38.913842 2647 tablet_bootstrap.cc:654] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:38.915064 2646 tablet_bootstrap.cc:654] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:38.915164 2649 tablet_bootstrap.cc:654] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:38.916487 2647 tablet_bootstrap.cc:492] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: No bootstrap required, opened a new log
I20260929 11:20:38.916661 2647 ts_tablet_manager.cc:1397] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 11:20:38.917212 2646 tablet_bootstrap.cc:492] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: No bootstrap required, opened a new log
I20260929 11:20:38.917351 2646 ts_tablet_manager.cc:1397] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 11:20:38.917462 2649 tablet_bootstrap.cc:492] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: No bootstrap required, opened a new log
I20260929 11:20:38.917315 2647 raft_consensus.cc:348] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:38.917598 2649 ts_tablet_manager.cc:1397] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:20:38.917606 2647 raft_consensus.cc:374] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:38.917699 2647 raft_consensus.cc:729] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9807cabe70404541b44412b7a58432e6, State: Initialized, Role: FOLLOWER
I20260929 11:20:38.917950 2647 consensus_queue.cc:260] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:38.918124 2646 raft_consensus.cc:348] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:38.918345 2646 raft_consensus.cc:374] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:38.918442 2646 raft_consensus.cc:729] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Initialized, Role: FOLLOWER
I20260929 11:20:38.918582 2649 raft_consensus.cc:348] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:38.918658 2646 consensus_queue.cc:260] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:38.918810 2649 raft_consensus.cc:374] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:38.919114 2653 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:38.919269 2646 ts_tablet_manager.cc:1428] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:20:38.919433 2653 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:38.918897 2649 raft_consensus.cc:729] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cf37ace4b34aed977f816b712e9559, State: Initialized, Role: FOLLOWER
I20260929 11:20:38.919836 2649 consensus_queue.cc:260] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:38.919015 2654 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:38.920346 2654 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:38.920475 2649 ts_tablet_manager.cc:1428] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:20:38.920284 2652 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:38.920571 2647 ts_tablet_manager.cc:1428] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 11:20:38.920653 2652 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:39.081458 2654 raft_consensus.cc:482] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:39.081629 2654 raft_consensus.cc:504] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.083120 2654 leader_election.cc:288] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903), 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041)
I20260929 11:20:39.090528 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.083701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52723 (local address 127.0.97.2:37041)
0929 11:20:39.083907 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:39.083913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:39.084639 (+ 726us) server_negotiation.cc:407] Connection header received
0929 11:20:39.084825 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:39.084830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:39.084910 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:39.085001 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:39.085636 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.086761 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.088670 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.089590 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.089741 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.089849 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:39.090177 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 11:20:39.090296 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:20:39.090627 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.083566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41739 (local address 127.0.97.1:44903)
0929 11:20:39.083789 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:39.083798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:39.084300 (+ 502us) server_negotiation.cc:407] Connection header received
0929 11:20:39.084607 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:39.084614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:39.084700 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:39.084803 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:39.085321 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.086486 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.088669 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.089636 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.089791 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.089896 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:39.090228 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 11:20:39.090362 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:20:39.090662 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.083440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37041 (local address 127.0.97.3:52723)
0929 11:20:39.084515 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:39.084538 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:39.084699 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:39.085038 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:39.085043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:39.085068 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:39.085398 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.085407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.086892 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.086897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.088497 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.088508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.089754 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.089760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.090009 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.090048 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 11:20:39.090153 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:20:39.090277 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":26,"thread_start_us":808,"threads_started":1}
I20260929 11:20:39.090935 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.083284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44903 (local address 127.0.97.3:41739)
0929 11:20:39.084179 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:39.084204 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:39.084419 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:39.084847 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:39.084851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:39.084873 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:39.085095 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.085109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.086623 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.086628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.088509 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.088520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.089849 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.089854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.090071 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.090093 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:20:39.090192 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:20:39.090314 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":58,"thread_start_us":709,"threads_started":1}
I20260929 11:20:39.091537 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aa4e54c037460faeb6b23142372cc7" candidate_uuid: "9807cabe70404541b44412b7a58432e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559" is_pre_election: true
I20260929 11:20:39.091555 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aa4e54c037460faeb6b23142372cc7" candidate_uuid: "9807cabe70404541b44412b7a58432e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca2e35d549f4cb1abae316482fd9743" is_pre_election: true
I20260929 11:20:39.091816 2299 raft_consensus.cc:2454] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9807cabe70404541b44412b7a58432e6 in term 0.
I20260929 11:20:39.091956 2441 raft_consensus.cc:2454] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9807cabe70404541b44412b7a58432e6 in term 0.
I20260929 11:20:39.092263 2508 leader_election.cc:302] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cf37ace4b34aed977f816b712e9559, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:39.092689 2654 raft_consensus.cc:2781] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:39.092818 2654 raft_consensus.cc:482] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:39.092919 2654 raft_consensus.cc:3037] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:39.095212 2654 raft_consensus.cc:504] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.095738 2654 leader_election.cc:288] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [CANDIDATE]: Term 1 election: Requested vote from peers 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903), 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041)
I20260929 11:20:39.096131 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aa4e54c037460faeb6b23142372cc7" candidate_uuid: "9807cabe70404541b44412b7a58432e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559"
I20260929 11:20:39.096197 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aa4e54c037460faeb6b23142372cc7" candidate_uuid: "9807cabe70404541b44412b7a58432e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca2e35d549f4cb1abae316482fd9743"
I20260929 11:20:39.096354 2299 raft_consensus.cc:3037] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:39.096371 2441 raft_consensus.cc:3037] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:39.099280 2299 raft_consensus.cc:2454] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9807cabe70404541b44412b7a58432e6 in term 1.
I20260929 11:20:39.099293 2441 raft_consensus.cc:2454] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9807cabe70404541b44412b7a58432e6 in term 1.
I20260929 11:20:39.099647 2508 leader_election.cc:302] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67cf37ace4b34aed977f816b712e9559, 8ca2e35d549f4cb1abae316482fd9743, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:39.099913 2654 raft_consensus.cc:2781] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:39.100304 2654 raft_consensus.cc:686] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9807cabe70404541b44412b7a58432e6, State: Running, Role: LEADER
I20260929 11:20:39.100811 2654 consensus_queue.cc:237] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.103925 2670 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: TxnStatusTablet state changed. Reason: New leader 9807cabe70404541b44412b7a58432e6. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.104259 2670 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:20:39.104990 2672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:20:39.105268 2672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:20:39.105937 2142 catalog_manager.cc:5495] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9807cabe70404541b44412b7a58432e6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } health_report { overall_health: HEALTHY } } }
I20260929 11:20:39.155030 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.149203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34339 (local address 127.0.0.1:56456)
0929 11:20:39.149479 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:39.149502 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:39.149663 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:39.150273 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:39.150278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:39.150289 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:39.150539 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.150564 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.151676 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.151682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.152722 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.152734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.153422 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.153428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.153609 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.153661 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.154503 (+ 842us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.154530 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:20:39.154612 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 11:20:39.154755 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 11:20:39.155067 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.149171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.97.3:34339)
0929 11:20:39.149359 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:39.149366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:39.149583 (+ 217us) server_negotiation.cc:407] Connection header received
0929 11:20:39.149937 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:39.149941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:39.150080 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:39.150258 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:39.150669 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.151537 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.152854 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.153277 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.153411 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.153839 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.154388 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.154546 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:39.154632 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 11:20:39.154711 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:20:39.163812 2441 raft_consensus.cc:1261] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Refusing update from remote peer 9807cabe70404541b44412b7a58432e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:39.163973 2299 raft_consensus.cc:1261] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Refusing update from remote peer 9807cabe70404541b44412b7a58432e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:39.164777 2670 consensus_queue.cc:1035] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:39.165114 2654 consensus_queue.cc:1035] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:39.177179 2653 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: TxnStatusTablet state changed. Reason: New leader 9807cabe70404541b44412b7a58432e6. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.177472 2653 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:39.179394 2675 mvcc.cc:205] Tried to move back new op lower bound from 7334628717205544960 to 7334628716960505856. Current Snapshot: MvccSnapshot[applied={T|T < 7334628717205544960}]
I20260929 11:20:39.180261 2676 mvcc.cc:205] Tried to move back new op lower bound from 7334628717205544960 to 7334628716960505856. Current Snapshot: MvccSnapshot[applied={T|T < 7334628717205544960}]
I20260929 11:20:39.180315 2654 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.180573 2654 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:20:39.183543 2653 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.183789 2653 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 8ca2e35d549f4cb1abae316482fd9743: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:39.187397 2652 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: TxnStatusTablet state changed. Reason: New leader 9807cabe70404541b44412b7a58432e6. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.187722 2652 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:39.188576 2670 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.188858 2670 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 9807cabe70404541b44412b7a58432e6: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:20:39.197144 2652 tablet_replica.cc:439] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9807cabe70404541b44412b7a58432e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } } }
I20260929 11:20:39.197363 2652 tablet_replica.cc:442] T 87aa4e54c037460faeb6b23142372cc7 P 67cf37ace4b34aed977f816b712e9559: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:20:39.201233 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.192754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.0.97.2:37041)
0929 11:20:39.192930 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:39.192937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:39.193539 (+ 602us) server_negotiation.cc:407] Connection header received
0929 11:20:39.193784 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:39.193792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:39.193944 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:39.194128 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:39.194646 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.196094 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.198844 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.199427 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.199510 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.200094 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.200644 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.200800 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:39.200877 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 11:20:39.200950 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:20:39.216785 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.210983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35055 (local address 127.0.97.3:34339)
0929 11:20:39.211190 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:39.211200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:39.211728 (+ 528us) server_negotiation.cc:407] Connection header received
0929 11:20:39.211969 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:39.211973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:39.212108 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:39.212181 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:39.212823 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.213659 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.215038 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.215467 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.215623 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.216039 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.216132 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.216238 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:39.216378 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 11:20:39.216449 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:20:39.216887 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.210973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34339 (local address 127.0.97.2:35055)
0929 11:20:39.211619 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:39.211640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:39.211796 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:39.212225 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:39.212231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:39.212274 (+ 43us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:39.212657 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.212670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.213805 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.213810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.214894 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.214905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.215647 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.215653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.215794 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.215842 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.216238 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.216251 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 11:20:39.216350 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:20:39.216476 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":335,"threads_started":1}
I20260929 11:20:39.226138 2299 raft_consensus.cc:1261] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Refusing update from remote peer 8ca2e35d549f4cb1abae316482fd9743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:39.226626 2582 raft_consensus.cc:1261] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Refusing update from remote peer 8ca2e35d549f4cb1abae316482fd9743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:39.226765 2657 consensus_queue.cc:1035] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:39.227351 2653 consensus_queue.cc:1035] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:39.234699 2678 mvcc.cc:205] Tried to move back new op lower bound from 7334628717463703552 to 7334628715751923712. Current Snapshot: MvccSnapshot[applied={T|T < 7334628717463703552}]
I20260929 11:20:39.268462 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.261301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51419 (local address 127.0.97.2:37041)
0929 11:20:39.261484 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:39.261490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:39.262262 (+ 772us) server_negotiation.cc:407] Connection header received
0929 11:20:39.262557 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:39.262563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:39.262646 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:39.262746 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:39.263490 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.264643 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.266226 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.266884 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.267029 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.267627 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.267759 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.267846 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:39.268055 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 11:20:39.268135 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:20:39.268631 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:39.261351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37041 (local address 127.0.97.3:51419)
0929 11:20:39.262148 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:39.262171 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:39.262333 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:39.262799 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:39.262804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:39.262818 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:39.263296 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.263311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.264767 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.264773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.266084 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:39.266097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:39.267100 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:39.267107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:39.267312 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:39.267395 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:39.267874 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:39.267887 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 11:20:39.268026 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 11:20:39.268173 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":402,"threads_started":1}
I20260929 11:20:39.313234 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:39.313532 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:39.317420 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Sending DeleteTablet for 3 replicas of tablet 3efdd8d34ced4c73a9793c471678395d
I20260929 11:20:39.318706 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 3efdd8d34ced4c73a9793c471678395d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53634
I20260929 11:20:39.319630 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 3efdd8d34ced4c73a9793c471678395d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46656
I20260929 11:20:39.319877 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:39.320497 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:39.322908 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 3efdd8d34ced4c73a9793c471678395d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56446
I20260929 11:20:39.327644 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 78130b408fbc4df98be969f86af06517 (DEFAULT_TABLE table=test-table [id=9457adb7889e44f8a14a3a87f66618fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:39.327972 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 78130b408fbc4df98be969f86af06517 (DEFAULT_TABLE table=test-table [id=9457adb7889e44f8a14a3a87f66618fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:39.328313 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78130b408fbc4df98be969f86af06517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:39.328500 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78130b408fbc4df98be969f86af06517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:39.333388 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 78130b408fbc4df98be969f86af06517 (DEFAULT_TABLE table=test-table [id=9457adb7889e44f8a14a3a87f66618fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:39.333986 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78130b408fbc4df98be969f86af06517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:39.335319 2649 tablet_bootstrap.cc:492] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Bootstrap starting.
I20260929 11:20:39.336967 2649 tablet_bootstrap.cc:654] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:39.338253 2647 tablet_bootstrap.cc:492] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Bootstrap starting.
I20260929 11:20:39.340302 2647 tablet_bootstrap.cc:654] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:39.340832 2646 tablet_bootstrap.cc:492] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Bootstrap starting.
I20260929 11:20:39.342833 2646 tablet_bootstrap.cc:654] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:39.362253 2697 tablet_replica.cc:330] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: stopping tablet replica
I20260929 11:20:39.363788 2646 tablet_bootstrap.cc:492] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: No bootstrap required, opened a new log
I20260929 11:20:39.364063 2646 ts_tablet_manager.cc:1397] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.001s
I20260929 11:20:39.365105 2697 raft_consensus.cc:2229] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:39.366667 2696 tablet_replica.cc:330] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: stopping tablet replica
I20260929 11:20:39.367290 2695 tablet_replica.cc:330] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: stopping tablet replica
I20260929 11:20:39.367874 2696 raft_consensus.cc:2229] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:39.368309 2697 raft_consensus.cc:2258] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:39.369001 2695 raft_consensus.cc:2229] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:20:39.369160 2696 raft_consensus.cc:2258] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:39.368808 2649 tablet_bootstrap.cc:492] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: No bootstrap required, opened a new log
I20260929 11:20:39.369535 2649 ts_tablet_manager.cc:1397] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.001s
I20260929 11:20:39.370229 2649 raft_consensus.cc:348] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.370468 2649 raft_consensus.cc:374] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:39.371083 2695 raft_consensus.cc:2258] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:39.372292 2697 ts_tablet_manager.cc:1905] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:39.366391 2646 raft_consensus.cc:348] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.372689 2646 raft_consensus.cc:374] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:39.372788 2646 raft_consensus.cc:729] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Initialized, Role: FOLLOWER
I20260929 11:20:39.372992 2649 raft_consensus.cc:729] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cf37ace4b34aed977f816b712e9559, State: Initialized, Role: FOLLOWER
I20260929 11:20:39.373255 2649 consensus_queue.cc:260] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.373884 2649 ts_tablet_manager.cc:1428] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260929 11:20:39.374080 2647 tablet_bootstrap.cc:492] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: No bootstrap required, opened a new log
I20260929 11:20:39.374158 2695 ts_tablet_manager.cc:1905] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:39.374243 2647 ts_tablet_manager.cc:1397] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.006s
I20260929 11:20:39.375005 2647 raft_consensus.cc:348] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.375236 2647 raft_consensus.cc:374] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:39.375316 2647 raft_consensus.cc:729] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9807cabe70404541b44412b7a58432e6, State: Initialized, Role: FOLLOWER
I20260929 11:20:39.375556 2647 consensus_queue.cc:260] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.376180 2647 ts_tablet_manager.cc:1428] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 11:20:39.376914 2646 consensus_queue.cc:260] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.377503 2646 ts_tablet_manager.cc:1428] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260929 11:20:39.378906 2696 ts_tablet_manager.cc:1905] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:39.387013 2696 ts_tablet_manager.cc:1918] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:39.387182 2696 log.cc:1196] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/3efdd8d34ced4c73a9793c471678395d
I20260929 11:20:39.387715 2696 ts_tablet_manager.cc:1939] T 3efdd8d34ced4c73a9793c471678395d P 67cf37ace4b34aed977f816b712e9559: Deleting consensus metadata
I20260929 11:20:39.389695 2127 catalog_manager.cc:4841] TS 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903): tablet 3efdd8d34ced4c73a9793c471678395d (table test-table [id=d85b33c7a83444dda300fb7a1a63d330]) successfully deleted
I20260929 11:20:39.391544 2697 ts_tablet_manager.cc:1918] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:39.391790 2697 log.cc:1196] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/3efdd8d34ced4c73a9793c471678395d
I20260929 11:20:39.392462 2697 ts_tablet_manager.cc:1939] T 3efdd8d34ced4c73a9793c471678395d P 9807cabe70404541b44412b7a58432e6: Deleting consensus metadata
I20260929 11:20:39.395345 2130 catalog_manager.cc:4841] TS 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339): tablet 3efdd8d34ced4c73a9793c471678395d (table test-table [id=d85b33c7a83444dda300fb7a1a63d330]) successfully deleted
I20260929 11:20:39.395821 2695 ts_tablet_manager.cc:1918] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:39.395987 2695 log.cc:1196] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/3efdd8d34ced4c73a9793c471678395d
I20260929 11:20:39.396443 2695 ts_tablet_manager.cc:1939] T 3efdd8d34ced4c73a9793c471678395d P 8ca2e35d549f4cb1abae316482fd9743: Deleting consensus metadata
I20260929 11:20:39.397796 2129 catalog_manager.cc:4841] TS 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041): tablet 3efdd8d34ced4c73a9793c471678395d (table test-table [id=d85b33c7a83444dda300fb7a1a63d330]) successfully deleted
I20260929 11:20:39.440217 2653 raft_consensus.cc:482] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:39.440433 2653 raft_consensus.cc:504] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.441227 2653 leader_election.cc:288] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903), 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:39.441948 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78130b408fbc4df98be969f86af06517" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559" is_pre_election: true
I20260929 11:20:39.441953 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78130b408fbc4df98be969f86af06517" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6" is_pre_election: true
I20260929 11:20:39.442188 2582 raft_consensus.cc:2454] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 0.
I20260929 11:20:39.442178 2299 raft_consensus.cc:2454] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 0.
I20260929 11:20:39.442667 2372 leader_election.cc:302] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca2e35d549f4cb1abae316482fd9743, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:39.442920 2653 raft_consensus.cc:2781] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:39.443040 2653 raft_consensus.cc:482] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:39.443138 2653 raft_consensus.cc:3037] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:39.445320 2653 raft_consensus.cc:504] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.445976 2653 leader_election.cc:288] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 election: Requested vote from peers 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903), 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:39.446414 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78130b408fbc4df98be969f86af06517" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559"
I20260929 11:20:39.446604 2299 raft_consensus.cc:3037] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:39.446884 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78130b408fbc4df98be969f86af06517" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6"
I20260929 11:20:39.447108 2582 raft_consensus.cc:3037] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:39.449532 2299 raft_consensus.cc:2454] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 1.
I20260929 11:20:39.450176 2582 raft_consensus.cc:2454] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 1.
I20260929 11:20:39.450629 2372 leader_election.cc:302] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca2e35d549f4cb1abae316482fd9743, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:39.451006 2653 raft_consensus.cc:2781] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:39.451179 2653 raft_consensus.cc:686] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Running, Role: LEADER
I20260929 11:20:39.451465 2653 consensus_queue.cc:237] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:39.454720 2142 catalog_manager.cc:5495] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ca2e35d549f4cb1abae316482fd9743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20260929 11:20:39.512494 2299 raft_consensus.cc:1261] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Refusing update from remote peer 8ca2e35d549f4cb1abae316482fd9743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:39.513162 2582 raft_consensus.cc:1261] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Refusing update from remote peer 8ca2e35d549f4cb1abae316482fd9743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:39.513173 2653 consensus_queue.cc:1035] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:39.513860 2657 consensus_queue.cc:1035] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:20:39.565310 2676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 11:20:41.169054 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:41.169272 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:41.173522 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Sending DeleteTablet for 3 replicas of tablet 78130b408fbc4df98be969f86af06517
I20260929 11:20:41.174243 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 78130b408fbc4df98be969f86af06517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46656
I20260929 11:20:41.174309 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 78130b408fbc4df98be969f86af06517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56446
I20260929 11:20:41.174465 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 78130b408fbc4df98be969f86af06517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53634
I20260929 11:20:41.186790 2732 tablet_replica.cc:330] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: stopping tablet replica
I20260929 11:20:41.186949 2731 tablet_replica.cc:330] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: stopping tablet replica
I20260929 11:20:41.187196 2731 raft_consensus.cc:2229] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:41.187302 2733 tablet_replica.cc:330] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: stopping tablet replica
I20260929 11:20:41.187472 2731 raft_consensus.cc:2258] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:41.187481 2733 raft_consensus.cc:2229] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:20:41.187853 2733 raft_consensus.cc:2258] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:41.188170 2732 raft_consensus.cc:2229] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:41.188390 2732 raft_consensus.cc:2258] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:41.188627 2731 ts_tablet_manager.cc:1905] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:41.189564 2732 ts_tablet_manager.cc:1905] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:41.193243 2733 ts_tablet_manager.cc:1905] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:41.194082 2731 ts_tablet_manager.cc:1918] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:20:41.194275 2731 log.cc:1196] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/78130b408fbc4df98be969f86af06517
I20260929 11:20:41.194818 2731 ts_tablet_manager.cc:1939] T 78130b408fbc4df98be969f86af06517 P 67cf37ace4b34aed977f816b712e9559: Deleting consensus metadata
I20260929 11:20:41.196022 2127 catalog_manager.cc:4841] TS 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903): tablet 78130b408fbc4df98be969f86af06517 (table test-table [id=9457adb7889e44f8a14a3a87f66618fc]) successfully deleted
I20260929 11:20:41.197095 2732 ts_tablet_manager.cc:1918] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:20:41.197258 2732 log.cc:1196] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/78130b408fbc4df98be969f86af06517
I20260929 11:20:41.197613 2732 ts_tablet_manager.cc:1939] T 78130b408fbc4df98be969f86af06517 P 9807cabe70404541b44412b7a58432e6: Deleting consensus metadata
I20260929 11:20:41.198443 2130 catalog_manager.cc:4841] TS 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339): tablet 78130b408fbc4df98be969f86af06517 (table test-table [id=9457adb7889e44f8a14a3a87f66618fc]) successfully deleted
I20260929 11:20:41.200461 2733 ts_tablet_manager.cc:1918] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:20:41.200650 2733 log.cc:1196] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/78130b408fbc4df98be969f86af06517
I20260929 11:20:41.201185 2733 ts_tablet_manager.cc:1939] T 78130b408fbc4df98be969f86af06517 P 8ca2e35d549f4cb1abae316482fd9743: Deleting consensus metadata
I20260929 11:20:41.202203 2129 catalog_manager.cc:4841] TS 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041): tablet 78130b408fbc4df98be969f86af06517 (table test-table [id=9457adb7889e44f8a14a3a87f66618fc]) successfully deleted
Sep 29 11:20:41 dist-test-slave-srst krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:20:41 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:41.230953 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:41.217404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.97.62:41101)
0929 11:20:41.217979 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:41.217985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:41.218007 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:20:41.218226 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:41.218238 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:41.218295 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:41.218405 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:41.219691 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:41.220453 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:41.221750 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:41.222147 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:41.222271 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:41.225928 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:41.225933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:41.225936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:41.225966 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:41.228441 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:41.229128 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:41.229133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:41.229138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:41.229264 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:41.229636 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:41.229643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:41.229651 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:41.229890 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:41.230142 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:41.230409 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:20:41.230626 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":292,"threads_started":1}
I20260929 11:20:41.234331 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:41.234951 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:41.241199 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 3df3e9f1e3b942829c6f2b303b19a2d3 (DEFAULT_TABLE table=test-table [id=36fd5f7e945249c58cc784b1218095a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:41.241590 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df3e9f1e3b942829c6f2b303b19a2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:41.241482 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 3df3e9f1e3b942829c6f2b303b19a2d3 (DEFAULT_TABLE table=test-table [id=36fd5f7e945249c58cc784b1218095a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:41.241508 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 3df3e9f1e3b942829c6f2b303b19a2d3 (DEFAULT_TABLE table=test-table [id=36fd5f7e945249c58cc784b1218095a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:41.241910 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df3e9f1e3b942829c6f2b303b19a2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:41.241910 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df3e9f1e3b942829c6f2b303b19a2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:41.246727 2742 tablet_bootstrap.cc:492] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Bootstrap starting.
I20260929 11:20:41.248818 2744 tablet_bootstrap.cc:492] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Bootstrap starting.
I20260929 11:20:41.248937 2742 tablet_bootstrap.cc:654] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:41.250700 2744 tablet_bootstrap.cc:654] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:41.252947 2742 tablet_bootstrap.cc:492] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: No bootstrap required, opened a new log
I20260929 11:20:41.253029 2743 tablet_bootstrap.cc:492] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Bootstrap starting.
I20260929 11:20:41.253079 2742 ts_tablet_manager.cc:1397] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 11:20:41.253201 2744 tablet_bootstrap.cc:492] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: No bootstrap required, opened a new log
I20260929 11:20:41.253329 2744 ts_tablet_manager.cc:1397] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 11:20:41.253820 2742 raft_consensus.cc:348] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.254060 2742 raft_consensus.cc:374] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:41.254160 2742 raft_consensus.cc:729] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9807cabe70404541b44412b7a58432e6, State: Initialized, Role: FOLLOWER
I20260929 11:20:41.254139 2744 raft_consensus.cc:348] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.254365 2744 raft_consensus.cc:374] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:41.254457 2744 raft_consensus.cc:729] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Initialized, Role: FOLLOWER
I20260929 11:20:41.254356 2742 consensus_queue.cc:260] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.254666 2743 tablet_bootstrap.cc:654] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:41.254698 2744 consensus_queue.cc:260] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.254894 2742 ts_tablet_manager.cc:1428] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:41.255214 2744 ts_tablet_manager.cc:1428] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:41.257568 2743 tablet_bootstrap.cc:492] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: No bootstrap required, opened a new log
I20260929 11:20:41.257725 2743 ts_tablet_manager.cc:1397] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 11:20:41.258445 2743 raft_consensus.cc:348] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.258714 2743 raft_consensus.cc:374] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:41.258821 2743 raft_consensus.cc:729] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cf37ace4b34aed977f816b712e9559, State: Initialized, Role: FOLLOWER
I20260929 11:20:41.259080 2743 consensus_queue.cc:260] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.259876 2743 ts_tablet_manager.cc:1428] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 11:20:41.490594 2710 raft_consensus.cc:482] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:41.490803 2710 raft_consensus.cc:504] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.491479 2710 leader_election.cc:288] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903), 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:41.492025 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df3e9f1e3b942829c6f2b303b19a2d3" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6" is_pre_election: true
I20260929 11:20:41.492034 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df3e9f1e3b942829c6f2b303b19a2d3" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559" is_pre_election: true
I20260929 11:20:41.492259 2582 raft_consensus.cc:2454] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 0.
I20260929 11:20:41.492280 2299 raft_consensus.cc:2454] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 0.
I20260929 11:20:41.492743 2372 leader_election.cc:302] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca2e35d549f4cb1abae316482fd9743, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:41.493021 2710 raft_consensus.cc:2781] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:41.493146 2710 raft_consensus.cc:482] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:41.493216 2710 raft_consensus.cc:3037] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:41.495061 2710 raft_consensus.cc:504] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.495512 2710 leader_election.cc:288] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 election: Requested vote from peers 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903), 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:41.495965 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df3e9f1e3b942829c6f2b303b19a2d3" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf37ace4b34aed977f816b712e9559"
I20260929 11:20:41.495991 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df3e9f1e3b942829c6f2b303b19a2d3" candidate_uuid: "8ca2e35d549f4cb1abae316482fd9743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6"
I20260929 11:20:41.496168 2299 raft_consensus.cc:3037] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:41.496193 2582 raft_consensus.cc:3037] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:41.499008 2299 raft_consensus.cc:2454] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 1.
I20260929 11:20:41.499009 2582 raft_consensus.cc:2454] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca2e35d549f4cb1abae316482fd9743 in term 1.
I20260929 11:20:41.499404 2369 leader_election.cc:302] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cf37ace4b34aed977f816b712e9559, 8ca2e35d549f4cb1abae316482fd9743; no voters:
I20260929 11:20:41.499662 2710 raft_consensus.cc:2781] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:41.499825 2710 raft_consensus.cc:686] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Running, Role: LEADER
I20260929 11:20:41.500092 2710 consensus_queue.cc:237] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.502449 2142 catalog_manager.cc:5495] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ca2e35d549f4cb1abae316482fd9743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20260929 11:20:41.555171 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:41.550141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.97.2:37041)
0929 11:20:41.550706 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:41.550710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:41.550727 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:20:41.550774 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:41.550778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:41.550884 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:41.550985 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:41.551395 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:41.552121 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:41.553451 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:41.553880 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:41.554012 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:41.554443 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:41.554547 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:41.554645 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:41.554790 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 11:20:41.554862 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":302,"threads_started":1}
I20260929 11:20:41.563380 2299 raft_consensus.cc:1261] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Refusing update from remote peer 8ca2e35d549f4cb1abae316482fd9743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:41.563405 2582 raft_consensus.cc:1261] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Refusing update from remote peer 8ca2e35d549f4cb1abae316482fd9743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:41.563925 2726 consensus_queue.cc:1035] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:41.564297 2710 consensus_queue.cc:1035] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:41.613624 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:41.613862 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:41.616626 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Sending DeleteTablet for 3 replicas of tablet 3df3e9f1e3b942829c6f2b303b19a2d3
I20260929 11:20:41.617282 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df3e9f1e3b942829c6f2b303b19a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46656
I20260929 11:20:41.617379 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df3e9f1e3b942829c6f2b303b19a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53634
I20260929 11:20:41.617411 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df3e9f1e3b942829c6f2b303b19a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56446
I20260929 11:20:41.617609 2731 tablet_replica.cc:330] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: stopping tablet replica
I20260929 11:20:41.617625 2733 tablet_replica.cc:330] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: stopping tablet replica
I20260929 11:20:41.617636 2732 tablet_replica.cc:330] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: stopping tablet replica
I20260929 11:20:41.617797 2731 raft_consensus.cc:2229] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:41.617797 2733 raft_consensus.cc:2229] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:20:41.617802 2732 raft_consensus.cc:2229] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:41.617947 2731 raft_consensus.cc:2258] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:41.617955 2732 raft_consensus.cc:2258] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:41.618083 2733 raft_consensus.cc:2258] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:41.618480 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 11:20:41.618960 2731 ts_tablet_manager.cc:1905] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:41.619083 2733 ts_tablet_manager.cc:1905] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 11:20:41.619117 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:41.619593 2732 ts_tablet_manager.cc:1905] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:41.623531 2731 ts_tablet_manager.cc:1918] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:41.623631 2731 log.cc:1196] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/3df3e9f1e3b942829c6f2b303b19a2d3
I20260929 11:20:41.623903 2731 ts_tablet_manager.cc:1939] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 67cf37ace4b34aed977f816b712e9559: Deleting consensus metadata
I20260929 11:20:41.624713 2127 catalog_manager.cc:4841] TS 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903): tablet 3df3e9f1e3b942829c6f2b303b19a2d3 (table test-table [id=36fd5f7e945249c58cc784b1218095a0]) successfully deleted
I20260929 11:20:41.625437 2733 ts_tablet_manager.cc:1918] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:41.625586 2733 log.cc:1196] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/3df3e9f1e3b942829c6f2b303b19a2d3
I20260929 11:20:41.626205 2733 ts_tablet_manager.cc:1939] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 8ca2e35d549f4cb1abae316482fd9743: Deleting consensus metadata
I20260929 11:20:41.626216 2732 ts_tablet_manager.cc:1918] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:20:41.626423 2732 log.cc:1196] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/3df3e9f1e3b942829c6f2b303b19a2d3
I20260929 11:20:41.626942 2732 ts_tablet_manager.cc:1939] T 3df3e9f1e3b942829c6f2b303b19a2d3 P 9807cabe70404541b44412b7a58432e6: Deleting consensus metadata
I20260929 11:20:41.627327 2129 catalog_manager.cc:4841] TS 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041): tablet 3df3e9f1e3b942829c6f2b303b19a2d3 (table test-table [id=36fd5f7e945249c58cc784b1218095a0]) successfully deleted
I20260929 11:20:41.628073 2130 catalog_manager.cc:4841] TS 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339): tablet 3df3e9f1e3b942829c6f2b303b19a2d3 (table test-table [id=36fd5f7e945249c58cc784b1218095a0]) successfully deleted
I20260929 11:20:41.630057 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 4c735ef1dee44489b3b82bd15a7926b2 (DEFAULT_TABLE table=test-table [id=718a9091972747969c9b5919c7f5b112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:41.630256 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 4c735ef1dee44489b3b82bd15a7926b2 (DEFAULT_TABLE table=test-table [id=718a9091972747969c9b5919c7f5b112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:41.630455 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c735ef1dee44489b3b82bd15a7926b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:41.630448 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 4c735ef1dee44489b3b82bd15a7926b2 (DEFAULT_TABLE table=test-table [id=718a9091972747969c9b5919c7f5b112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:41.630621 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c735ef1dee44489b3b82bd15a7926b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:41.630831 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c735ef1dee44489b3b82bd15a7926b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:41.635362 2742 tablet_bootstrap.cc:492] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Bootstrap starting.
I20260929 11:20:41.636502 2744 tablet_bootstrap.cc:492] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Bootstrap starting.
I20260929 11:20:41.636974 2742 tablet_bootstrap.cc:654] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:41.636969 2743 tablet_bootstrap.cc:492] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Bootstrap starting.
I20260929 11:20:41.638180 2744 tablet_bootstrap.cc:654] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:41.638610 2743 tablet_bootstrap.cc:654] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:41.639153 2742 tablet_bootstrap.cc:492] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: No bootstrap required, opened a new log
I20260929 11:20:41.639379 2742 ts_tablet_manager.cc:1397] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 11:20:41.640156 2742 raft_consensus.cc:348] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.640389 2742 raft_consensus.cc:374] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:41.640483 2742 raft_consensus.cc:729] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9807cabe70404541b44412b7a58432e6, State: Initialized, Role: FOLLOWER
I20260929 11:20:41.640755 2742 consensus_queue.cc:260] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.641037 2743 tablet_bootstrap.cc:492] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: No bootstrap required, opened a new log
I20260929 11:20:41.641043 2744 tablet_bootstrap.cc:492] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: No bootstrap required, opened a new log
I20260929 11:20:41.641184 2743 ts_tablet_manager.cc:1397] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:20:41.641196 2744 ts_tablet_manager.cc:1397] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 11:20:41.641470 2742 ts_tablet_manager.cc:1428] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:41.641883 2744 raft_consensus.cc:348] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.641929 2743 raft_consensus.cc:348] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.642130 2744 raft_consensus.cc:374] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:41.642158 2743 raft_consensus.cc:374] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:41.642232 2743 raft_consensus.cc:729] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cf37ace4b34aed977f816b712e9559, State: Initialized, Role: FOLLOWER
I20260929 11:20:41.642231 2744 raft_consensus.cc:729] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca2e35d549f4cb1abae316482fd9743, State: Initialized, Role: FOLLOWER
I20260929 11:20:41.642452 2743 consensus_queue.cc:260] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.642657 2744 consensus_queue.cc:260] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.643560 2743 ts_tablet_manager.cc:1428] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:41.643991 2744 ts_tablet_manager.cc:1428] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:20:41.996326 2748 raft_consensus.cc:482] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:20:41.996630 2748 raft_consensus.cc:504] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:41.998711 2748 leader_election.cc:288] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041), 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:42.013597 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:41.999289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60589 (local address 127.0.97.3:34339)
0929 11:20:42.001127 (+ 1838us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:42.001134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:42.008053 (+ 6919us) server_negotiation.cc:407] Connection header received
0929 11:20:42.008264 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:42.008271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:42.008347 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:42.008419 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:42.009225 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.009966 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.011773 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.012527 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.012758 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.012895 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:42.013152 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 11:20:42.013280 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1700,"thread_start_us":2092,"threads_started":1}
I20260929 11:20:42.013669 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:41.999020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34339 (local address 127.0.97.1:60589)
0929 11:20:42.007942 (+ 8922us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:42.007959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:42.008099 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:42.008687 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:42.008691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:42.008719 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:42.009042 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.009050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.010107 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.010114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.011604 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.011616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.012801 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.012808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.012996 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.013025 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:20:42.013128 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 11:20:42.013265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8758,"mutex_wait_us":2,"thread_start_us":8233,"threads_started":1}
I20260929 11:20:42.014657 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:41.999435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36663 (local address 127.0.97.2:37041)
0929 11:20:42.007210 (+ 7775us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:42.007219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:42.007677 (+ 458us) server_negotiation.cc:407] Connection header received
0929 11:20:42.007970 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:42.007976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:42.008052 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:42.008139 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:42.008656 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.009778 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.012263 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.013499 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.013677 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.013778 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:42.014290 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 11:20:42.014420 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7585}
I20260929 11:20:42.014564 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c735ef1dee44489b3b82bd15a7926b2" candidate_uuid: "67cf37ace4b34aed977f816b712e9559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6" is_pre_election: true
I20260929 11:20:42.014686 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:41.998985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37041 (local address 127.0.97.1:36663)
0929 11:20:42.007552 (+ 8567us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:42.007576 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:42.007783 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:42.008168 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:42.008172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:42.008205 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:42.008493 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.008507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.009961 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.009969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.012096 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.012110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.013922 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.013928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.014095 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.014124 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:20:42.014236 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 11:20:42.014355 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8399,"thread_start_us":8555,"threads_started":1}
I20260929 11:20:42.014837 2582 raft_consensus.cc:2454] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67cf37ace4b34aed977f816b712e9559 in term 0.
I20260929 11:20:42.015354 2233 leader_election.cc:302] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cf37ace4b34aed977f816b712e9559, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:42.015637 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c735ef1dee44489b3b82bd15a7926b2" candidate_uuid: "67cf37ace4b34aed977f816b712e9559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca2e35d549f4cb1abae316482fd9743" is_pre_election: true
I20260929 11:20:42.015756 2748 raft_consensus.cc:2781] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:20:42.015887 2748 raft_consensus.cc:482] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:20:42.015902 2440 raft_consensus.cc:2454] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67cf37ace4b34aed977f816b712e9559 in term 0.
I20260929 11:20:42.015997 2748 raft_consensus.cc:3037] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:42.019448 2748 raft_consensus.cc:504] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:42.020138 2748 leader_election.cc:288] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [CANDIDATE]: Term 1 election: Requested vote from peers 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041), 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339)
I20260929 11:20:42.020514 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c735ef1dee44489b3b82bd15a7926b2" candidate_uuid: "67cf37ace4b34aed977f816b712e9559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca2e35d549f4cb1abae316482fd9743"
I20260929 11:20:42.020741 2440 raft_consensus.cc:3037] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:42.020635 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c735ef1dee44489b3b82bd15a7926b2" candidate_uuid: "67cf37ace4b34aed977f816b712e9559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9807cabe70404541b44412b7a58432e6"
I20260929 11:20:42.020891 2582 raft_consensus.cc:3037] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:42.022823 2582 raft_consensus.cc:2454] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67cf37ace4b34aed977f816b712e9559 in term 1.
I20260929 11:20:42.023181 2233 leader_election.cc:302] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cf37ace4b34aed977f816b712e9559, 9807cabe70404541b44412b7a58432e6; no voters:
I20260929 11:20:42.023456 2748 raft_consensus.cc:2781] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:42.023572 2440 raft_consensus.cc:2454] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67cf37ace4b34aed977f816b712e9559 in term 1.
I20260929 11:20:42.030761 2748 raft_consensus.cc:686] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 1 LEADER]: Becoming Leader. State: Replica: 67cf37ace4b34aed977f816b712e9559, State: Running, Role: LEADER
I20260929 11:20:42.031170 2748 consensus_queue.cc:237] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } }
I20260929 11:20:42.034288 2143 catalog_manager.cc:5495] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67cf37ace4b34aed977f816b712e9559 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "67cf37ace4b34aed977f816b712e9559" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf37ace4b34aed977f816b712e9559" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20260929 11:20:42.109433 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:42.103645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.0.97.1:44903)
0929 11:20:42.104259 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:42.104264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:42.104284 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:20:42.104391 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:42.104394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:42.104463 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:42.104661 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:42.105109 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.105951 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.107395 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.107782 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.107886 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.108288 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:42.108899 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:42.109010 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:42.109110 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 11:20:42.109167 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":328,"threads_started":1}
I20260929 11:20:42.123873 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:42.118613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55455 (local address 127.0.97.3:34339)
0929 11:20:42.118765 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:42.118769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:42.119302 (+ 533us) server_negotiation.cc:407] Connection header received
0929 11:20:42.119509 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:42.119513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:42.119573 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:42.119674 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:42.120140 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.120859 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.122251 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.122643 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.122777 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.123176 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:42.123269 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:42.123362 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:42.123493 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 11:20:42.123566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:20:42.123962 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:42.118611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34339 (local address 127.0.97.1:55455)
0929 11:20:42.119201 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:42.119219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:42.119371 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:42.119679 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:42.119684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:42.119695 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:42.119981 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.119989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.120983 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.120988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.122048 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.122059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.122806 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.122811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.122948 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.123002 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:42.123364 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:42.123377 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 11:20:42.123475 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 11:20:42.123596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":299,"threads_started":1}
I20260929 11:20:42.132023 2440 raft_consensus.cc:1261] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Refusing update from remote peer 67cf37ace4b34aed977f816b712e9559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:42.132099 2582 raft_consensus.cc:1261] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Refusing update from remote peer 67cf37ace4b34aed977f816b712e9559: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:20:42.132737 2763 consensus_queue.cc:1035] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9807cabe70404541b44412b7a58432e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:42.133073 2748 consensus_queue.cc:1035] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca2e35d549f4cb1abae316482fd9743" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:20:42.158713 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:42.153369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56427 (local address 127.0.97.1:44903)
0929 11:20:42.153696 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:42.153703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:42.153963 (+ 260us) server_negotiation.cc:407] Connection header received
0929 11:20:42.154151 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:42.154156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:42.154230 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:42.154319 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:20:42.154706 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.155499 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.157097 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.157551 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.157694 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.158052 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:20:42.158142 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:20:42.158237 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:42.158341 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 11:20:42.158412 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:20:42.158771 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:42.153359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44903 (local address 127.0.97.3:56427)
0929 11:20:42.153867 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:42.153883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:42.154023 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:42.154334 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:42.154338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:42.154350 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:20:42.154556 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.154564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.155625 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.155631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.156966 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:42.156977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:42.157698 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:42.157704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:42.157862 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:42.157895 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:20:42.158236 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:20:42.158249 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 11:20:42.158325 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 11:20:42.158459 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
W20260929 11:20:42.159798 2715 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 11:20:43.753782 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:43.754029 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:20:43.757887 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b932712a13224553a7ee3eaab22ea73e: Sending DeleteTablet for 3 replicas of tablet 4c735ef1dee44489b3b82bd15a7926b2
I20260929 11:20:43.758602 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c735ef1dee44489b3b82bd15a7926b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46656
I20260929 11:20:43.758783 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c735ef1dee44489b3b82bd15a7926b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53634
I20260929 11:20:43.758811 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c735ef1dee44489b3b82bd15a7926b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:20:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56446
I20260929 11:20:43.759698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2214
I20260929 11:20:43.771019 2803 tablet_replica.cc:330] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: stopping tablet replica
I20260929 11:20:43.771071 2801 tablet_replica.cc:330] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: stopping tablet replica
I20260929 11:20:43.771356 2802 tablet_replica.cc:330] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: stopping tablet replica
I20260929 11:20:43.771382 2803 raft_consensus.cc:2229] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:43.771435 2801 raft_consensus.cc:2229] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:20:43.771654 2802 raft_consensus.cc:2229] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:20:43.771728 2803 raft_consensus.cc:2258] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:43.771941 2802 raft_consensus.cc:2258] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:43.771944 2801 raft_consensus.cc:2258] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:20:43.773166 2803 ts_tablet_manager.cc:1905] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:43.773167 2801 ts_tablet_manager.cc:1905] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:43.773633 2802 ts_tablet_manager.cc:1905] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:20:43.779690 2803 ts_tablet_manager.cc:1918] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:20:43.779690 2801 ts_tablet_manager.cc:1918] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:20:43.779843 2801 log.cc:1196] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/4c735ef1dee44489b3b82bd15a7926b2
I20260929 11:20:43.779876 2803 log.cc:1196] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/4c735ef1dee44489b3b82bd15a7926b2
I20260929 11:20:43.780326 2801 ts_tablet_manager.cc:1939] T 4c735ef1dee44489b3b82bd15a7926b2 P 67cf37ace4b34aed977f816b712e9559: Deleting consensus metadata
I20260929 11:20:43.780351 2803 ts_tablet_manager.cc:1939] T 4c735ef1dee44489b3b82bd15a7926b2 P 8ca2e35d549f4cb1abae316482fd9743: Deleting consensus metadata
I20260929 11:20:43.780575 2802 ts_tablet_manager.cc:1918] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:20:43.780740 2802 log.cc:1196] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/4c735ef1dee44489b3b82bd15a7926b2
I20260929 11:20:43.781097 2802 ts_tablet_manager.cc:1939] T 4c735ef1dee44489b3b82bd15a7926b2 P 9807cabe70404541b44412b7a58432e6: Deleting consensus metadata
I20260929 11:20:43.781337 2127 catalog_manager.cc:4841] TS 67cf37ace4b34aed977f816b712e9559 (127.0.97.1:44903): tablet 4c735ef1dee44489b3b82bd15a7926b2 (table test-table [id=718a9091972747969c9b5919c7f5b112]) successfully deleted
I20260929 11:20:43.781370 2129 catalog_manager.cc:4841] TS 8ca2e35d549f4cb1abae316482fd9743 (127.0.97.2:37041): tablet 4c735ef1dee44489b3b82bd15a7926b2 (table test-table [id=718a9091972747969c9b5919c7f5b112]) successfully deleted
I20260929 11:20:43.782308 2130 catalog_manager.cc:4841] TS 9807cabe70404541b44412b7a58432e6 (127.0.97.3:34339): tablet 4c735ef1dee44489b3b82bd15a7926b2 (table test-table [id=718a9091972747969c9b5919c7f5b112]) successfully deleted
Sep 29 11:20:43 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:43.792200 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:43.764847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.1:44903)
0929 11:20:43.772739 (+ 7892us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:43.772747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:43.772770 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:20:43.772826 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:43.772836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:43.772889 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:43.773011 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:43.774526 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:43.775684 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:43.777501 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:43.778200 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:43.778322 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:43.787072 (+ 8750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:43.787086 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:43.787102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:43.787134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:43.790066 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:43.790644 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:43.790651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:43.790653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:43.790764 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:43.791054 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:43.791056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:43.791058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:43.791369 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:43.791576 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:43.791774 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 11:20:43.791925 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7758,"thread_start_us":7599,"threads_started":1}
W20260929 11:20:43.792873 2344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740
W20260929 11:20:43.793457 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:43.793637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 2214
I20260929 11:20:43.815303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2353
Sep 29 11:20:43 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:20:43.828933 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:43.816055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.0.97.2:37041)
0929 11:20:43.816543 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:43.816547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:43.816563 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:20:43.816658 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:43.816661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:43.816718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:43.816805 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:43.817931 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:43.818612 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:43.819740 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:43.820142 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:43.820236 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:43.824028 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:43.824040 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:43.824056 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:43.824086 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:43.826727 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:43.827304 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:43.827310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:43.827312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:43.827434 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:43.827739 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:43.827741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:43.827743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:43.828087 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:43.828317 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:43.828477 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 11:20:43.828681 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
W20260929 11:20:43.829620 2483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49536
W20260929 11:20:43.830259 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:43.830382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 2353
I20260929 11:20:43.852396 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2492
Sep 29 11:20:43 dist-test-slave-srst krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:20:43.866798 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:43.853215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.97.3:34339)
0929 11:20:43.853702 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:43.853706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:43.853721 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:20:43.853770 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:43.853774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:43.853827 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:43.853914 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:43.855147 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:43.855864 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:43.857172 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:43.857557 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:43.857645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:43.861618 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:43.861635 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:43.861656 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:43.861699 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:43.864458 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:43.865170 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:43.865176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:43.865178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:43.865308 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:43.865642 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:43.865645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:43.865647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:43.865970 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:43.866183 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:43.866394 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 11:20:43.866540 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
W20260929 11:20:43.867578 2622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658
W20260929 11:20:43.868242 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2492: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:43.868412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 2492
I20260929 11:20:43.889467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2111
I20260929 11:20:43.890631 2183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:44.004611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 2111
2026-09-29T11:20:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8681 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:44 dist-test-slave-srst krb5kdc[2811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:44 dist-test-slave-srst krb5kdc[2811](info): set up 2 sockets
Sep 29 11:20:44 dist-test-slave-srst krb5kdc[2811](info): commencing operation
krb5kdc: starting...
W20260929 11:20:45.088886 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:45 dist-test-slave-srst krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:45.268899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:45.584802 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:45.585060 2824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:45.585258 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:45.592087 2824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:45.592214 2824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:45.592299 2824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:45.592399 2824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:45.592489 2824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:45.601770 2824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:45.602958 2824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:45.604192 2824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:45.615479 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:45.615559 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:45.615520 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:45.616441 2824 server_base.cc:1034] running on GCE node
I20260929 11:20:45.617190 2824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:45.618511 2824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:45.619704 2824 hybrid_clock.cc:648] HybridClock initialized: now 1790680845619651 us; error 52 us; skew 500 ppm
I20260929 11:20:45.620035 2824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:45 dist-test-slave-srst krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:45.622985 2824 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:45.624344 2824 webserver.cc:460] Webserver started at http://127.0.97.62:40459/ using document root <none> and password file <none>
I20260929 11:20:45.625038 2824 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:45.625169 2824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:45.625480 2824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:45.628193 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "942ac8b9fa9945f19abcedfa835d0253"
format_stamp: "Formatted at 2026-09-29 11:20:45 on dist-test-slave-srst"
server_key: "0f7e8270f85d2c359f81a32b9f6ec8e2"
server_key_iv: "ea27ddb95aeb2d21a4d8f8eb8440f036"
server_key_version: "encryptionkey@0"
I20260929 11:20:45.628980 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "942ac8b9fa9945f19abcedfa835d0253"
format_stamp: "Formatted at 2026-09-29 11:20:45 on dist-test-slave-srst"
server_key: "0f7e8270f85d2c359f81a32b9f6ec8e2"
server_key_iv: "ea27ddb95aeb2d21a4d8f8eb8440f036"
server_key_version: "encryptionkey@0"
I20260929 11:20:45.633720 2824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:20:45.636893 2839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:45.638079 2824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 11:20:45.638245 2824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "942ac8b9fa9945f19abcedfa835d0253"
format_stamp: "Formatted at 2026-09-29 11:20:45 on dist-test-slave-srst"
server_key: "0f7e8270f85d2c359f81a32b9f6ec8e2"
server_key_iv: "ea27ddb95aeb2d21a4d8f8eb8440f036"
server_key_version: "encryptionkey@0"
I20260929 11:20:45.638445 2824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:45.654909 2824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:45.658599 2824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:45.658869 2824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:45.676452 2824 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44823
I20260929 11:20:45.676486 2890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44823 every 8 connection(s)
I20260929 11:20:45.677731 2824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:45.681298 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:45.685140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 2824
I20260929 11:20:45.685377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:45.686124 388 external_mini_cluster.cc:1428] Setting key 2554a85ad277061fb5ab8901b544e2c8
I20260929 11:20:45.694422 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253: Bootstrap starting.
I20260929 11:20:45.698042 2891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:45.699281 2891 log.cc:824] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:20:45 dist-test-slave-srst krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:45.702137 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253: No bootstrap required, opened a new log
I20260929 11:20:45.708489 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:45.688104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.0.97.62:44823)
0929 11:20:45.688871 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:45.688882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:45.688922 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:20:45.690435 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:45.690456 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:45.690832 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:45.691143 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:45.692890 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:45.694193 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:45.695922 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:45.696561 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:45.696765 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:45.701424 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:45.701432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:45.701446 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:45.701479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:45.704482 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:45.705197 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:45.705206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:45.705212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:45.705374 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:45.705761 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:45.705765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:45.705768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:45.706300 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:45.706616 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:45.707051 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 11:20:45.707448 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":362,"threads_started":1}
I20260929 11:20:45.710024 2891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942ac8b9fa9945f19abcedfa835d0253" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44823 } }
I20260929 11:20:45.710423 2891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:45.710542 2891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942ac8b9fa9945f19abcedfa835d0253, State: Initialized, Role: FOLLOWER
I20260929 11:20:45.711164 2891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942ac8b9fa9945f19abcedfa835d0253" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44823 } }
I20260929 11:20:45.711441 2891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:45.711557 2891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:45.711671 2891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:45.713760 2891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942ac8b9fa9945f19abcedfa835d0253" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44823 } }
I20260929 11:20:45.714177 2891 leader_election.cc:302] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942ac8b9fa9945f19abcedfa835d0253; no voters:
I20260929 11:20:45.714665 2891 leader_election.cc:288] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:45.714808 2896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:45.715476 2896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [term 1 LEADER]: Becoming Leader. State: Replica: 942ac8b9fa9945f19abcedfa835d0253, State: Running, Role: LEADER
I20260929 11:20:45.715961 2896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942ac8b9fa9945f19abcedfa835d0253" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44823 } }
I20260929 11:20:45.716609 2891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:45.718883 2898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942ac8b9fa9945f19abcedfa835d0253. Latest consensus state: current_term: 1 leader_uuid: "942ac8b9fa9945f19abcedfa835d0253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942ac8b9fa9945f19abcedfa835d0253" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44823 } } }
I20260929 11:20:45.719347 2898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:45.720731 2897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942ac8b9fa9945f19abcedfa835d0253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942ac8b9fa9945f19abcedfa835d0253" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44823 } } }
I20260929 11:20:45.721081 2905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:45.721186 2897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:45.725538 2905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:45.734357 2905 catalog_manager.cc:1347] Generated new cluster ID: be9d827125254631a8c8b9c7b4c6b0c7
I20260929 11:20:45.734489 2905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:45.742951 2905 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:45.744169 2905 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:45.753044 2905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942ac8b9fa9945f19abcedfa835d0253: Generated new TSK 0
I20260929 11:20:45.753825 2905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:45.804672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44823
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:46.133242 2917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:46.133536 2917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:46.133695 2917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:46.140527 2917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:46.140698 2917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:46.140865 2917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:46.150489 2917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:46.151726 2917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:46.153026 2917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:46.165629 2923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:46.165596 2922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:46.165706 2925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:46.166328 2917 server_base.cc:1034] running on GCE node
I20260929 11:20:46.166932 2917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:46.167866 2917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:46.169093 2917 hybrid_clock.cc:648] HybridClock initialized: now 1790680846169042 us; error 71 us; skew 500 ppm
I20260929 11:20:46.169400 2917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:46 dist-test-slave-srst krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:46.172405 2917 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:46.173746 2917 webserver.cc:460] Webserver started at http://127.0.97.1:35719/ using document root <none> and password file <none>
I20260929 11:20:46.174345 2917 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:46.174448 2917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:46.174679 2917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:46.177381 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "0cbec4d0f16f480aa7bfd755b3786c6b"
format_stamp: "Formatted at 2026-09-29 11:20:46 on dist-test-slave-srst"
server_key: "811f8202d348f4e3b517001e3e36d4f8"
server_key_iv: "8f4376ffdf310940a603d313ae754a7b"
server_key_version: "encryptionkey@0"
I20260929 11:20:46.178099 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0cbec4d0f16f480aa7bfd755b3786c6b"
format_stamp: "Formatted at 2026-09-29 11:20:46 on dist-test-slave-srst"
server_key: "811f8202d348f4e3b517001e3e36d4f8"
server_key_iv: "8f4376ffdf310940a603d313ae754a7b"
server_key_version: "encryptionkey@0"
I20260929 11:20:46.182832 2917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 11:20:46.185979 2932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:46.187381 2917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:20:46.187580 2917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "0cbec4d0f16f480aa7bfd755b3786c6b"
format_stamp: "Formatted at 2026-09-29 11:20:46 on dist-test-slave-srst"
server_key: "811f8202d348f4e3b517001e3e36d4f8"
server_key_iv: "8f4376ffdf310940a603d313ae754a7b"
server_key_version: "encryptionkey@0"
I20260929 11:20:46.187767 2917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:46.204481 2917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:46.208585 2917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:46.208913 2917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:46.209908 2917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:46.211510 2917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:46.211620 2917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:46.211784 2917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:46.211866 2917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:46.246590 2917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34527
I20260929 11:20:46.246624 3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34527 every 8 connection(s)
I20260929 11:20:46.248056 2917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:46.253402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 2917
I20260929 11:20:46.253618 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:46.254354 388 external_mini_cluster.cc:1428] Setting key ab35a828f962dec99f3d2a34141cfed2
Sep 29 11:20:46 dist-test-slave-srst krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:46.269940 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:46.250883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34817 (local address 127.0.97.62:44823)
0929 11:20:46.251351 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:46.251358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:46.252266 (+ 908us) server_negotiation.cc:407] Connection header received
0929 11:20:46.254730 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:46.254736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:46.254826 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:46.254958 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:46.256742 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.257912 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.259430 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.260156 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.260309 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:46.264037 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:46.264042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:46.264045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:46.264093 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:46.266966 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:46.267639 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:46.267644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:46.267647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:46.267772 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:46.268281 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:46.268284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:46.268286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:46.268533 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:46.268720 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:46.269389 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 11:20:46.269590 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260929 11:20:46.270859 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:46.251441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44823 (local address 127.0.97.1:34817)
0929 11:20:46.252108 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:46.252144 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:46.254434 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:46.255299 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:46.255307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:46.255783 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:46.256533 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:46.256550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.258071 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.258075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:46.259288 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:46.259298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.260360 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.260364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:46.260547 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:46.261304 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:46.261344 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:46.263824 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:46.267101 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:46.267108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:46.267124 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:46.267516 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:46.267900 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:46.267904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:46.267908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:46.268131 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:46.268695 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:46.268703 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:46.269033 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 11:20:46.269286 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 11:20:46.269782 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":260,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:20:46.272337 3045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44823
I20260929 11:20:46.272709 3045 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:46.273425 3045 heartbeater.cc:511] Master 127.0.97.62:44823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 11:20:46.275381 2856 ts_manager.cc:194] Registered new tserver with Master: 0cbec4d0f16f480aa7bfd755b3786c6b (127.0.97.1:34527)
I20260929 11:20:46.276839 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:46.296306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44823
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:46.609457 3051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:46.609747 3051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:46.609903 3051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:46.616880 3051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:46.617002 3051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:46.617205 3051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:46.627091 3051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:46.628341 3051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:46.629608 3051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:46.642287 3057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:46.642346 3059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:46.642381 3056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:46.643522 3051 server_base.cc:1034] running on GCE node
I20260929 11:20:46.644160 3051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:46.645150 3051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:46.646380 3051 hybrid_clock.cc:648] HybridClock initialized: now 1790680846646345 us; error 57 us; skew 500 ppm
I20260929 11:20:46.646718 3051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:46 dist-test-slave-srst krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:46.649962 3051 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:46.651423 3051 webserver.cc:460] Webserver started at http://127.0.97.2:45957/ using document root <none> and password file <none>
I20260929 11:20:46.652102 3051 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:46.652199 3051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:46.652465 3051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:46.655239 3051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "e83e404a4c1e483ca9fda51346015053"
format_stamp: "Formatted at 2026-09-29 11:20:46 on dist-test-slave-srst"
server_key: "a409283231cc81996968b6fc5b80f918"
server_key_iv: "a795e3dd24d574e9038f5cdc4db9b62f"
server_key_version: "encryptionkey@0"
I20260929 11:20:46.655910 3051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e83e404a4c1e483ca9fda51346015053"
format_stamp: "Formatted at 2026-09-29 11:20:46 on dist-test-slave-srst"
server_key: "a409283231cc81996968b6fc5b80f918"
server_key_iv: "a795e3dd24d574e9038f5cdc4db9b62f"
server_key_version: "encryptionkey@0"
I20260929 11:20:46.660914 3051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 11:20:46.664178 3066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:46.665573 3051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:46.665723 3051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "e83e404a4c1e483ca9fda51346015053"
format_stamp: "Formatted at 2026-09-29 11:20:46 on dist-test-slave-srst"
server_key: "a409283231cc81996968b6fc5b80f918"
server_key_iv: "a795e3dd24d574e9038f5cdc4db9b62f"
server_key_version: "encryptionkey@0"
I20260929 11:20:46.665897 3051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:46.681782 3051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:46.686740 3051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:46.687042 3051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:46.688089 3051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:46.689633 3051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:46.689749 3051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:46.689894 3051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:46.689966 3051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:46.723023 3051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35781
I20260929 11:20:46.723105 3178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35781 every 8 connection(s)
I20260929 11:20:46.724299 3051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:46.733157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3051
I20260929 11:20:46.733366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:46.733768 388 external_mini_cluster.cc:1428] Setting key 8e2302181be6abb343429cd671aad332
Sep 29 11:20:46 dist-test-slave-srst krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:46.744258 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:46.727006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53053 (local address 127.0.97.62:44823)
0929 11:20:46.727181 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:46.727185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:46.728280 (+ 1095us) server_negotiation.cc:407] Connection header received
0929 11:20:46.730282 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:46.730287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:46.730404 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:46.730521 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:46.731984 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.732777 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.734655 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.735117 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.735253 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:46.738488 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:46.738493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:46.738496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:46.738527 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:46.741108 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:46.742083 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:46.742086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:46.742088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:46.742201 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:46.742657 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:46.742660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:46.742662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:46.742957 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:46.743073 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:46.743869 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 11:20:46.744017 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:20:46.745888 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:46.727418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44823 (local address 127.0.97.2:53053)
0929 11:20:46.728097 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:46.728132 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:46.730021 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:46.730751 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:46.730759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:46.731257 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:46.731822 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:46.731839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.732956 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.732960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:46.734536 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:46.734548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:46.735331 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:46.735338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:46.735553 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:46.736197 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:46.736218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:46.738269 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:46.741292 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:46.741303 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:46.741323 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:46.741980 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:46.742310 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:46.742318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:46.742325 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:46.742556 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:46.743084 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:46.743094 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:46.743512 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 11:20:46.743804 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 11:20:46.744368 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
I20260929 11:20:46.747407 3179 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44823
I20260929 11:20:46.747757 3179 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:46.748467 3179 heartbeater.cc:511] Master 127.0.97.62:44823 requested a full tablet report, sending...
I20260929 11:20:46.750062 2856 ts_manager.cc:194] Registered new tserver with Master: e83e404a4c1e483ca9fda51346015053 (127.0.97.2:35781)
I20260929 11:20:46.751279 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53053
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:46.776206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44823
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:47.092900 3184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:47.093173 3184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:47.093266 3184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:47.100219 3184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:47.100327 3184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:47.100438 3184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:47.109701 3184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:47.110934 3184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:47.112255 3184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:47.125583 3192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:47.125615 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:47.125540 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:47.126274 3184 server_base.cc:1034] running on GCE node
I20260929 11:20:47.126803 3184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:47.127671 3184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:47.128859 3184 hybrid_clock.cc:648] HybridClock initialized: now 1790680847128813 us; error 34 us; skew 500 ppm
I20260929 11:20:47.129154 3184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:47 dist-test-slave-srst krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:47.132661 3184 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:47.134063 3184 webserver.cc:460] Webserver started at http://127.0.97.3:37833/ using document root <none> and password file <none>
I20260929 11:20:47.134806 3184 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:47.134938 3184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:47.135244 3184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:47.138111 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "ae600dff18f24370bb28ea8d0a681c12"
format_stamp: "Formatted at 2026-09-29 11:20:47 on dist-test-slave-srst"
server_key: "57d4e7a7ed36bfde184a759ed4d5289c"
server_key_iv: "5c48b53a4352aa07833f783dccf15a1a"
server_key_version: "encryptionkey@0"
I20260929 11:20:47.138846 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ae600dff18f24370bb28ea8d0a681c12"
format_stamp: "Formatted at 2026-09-29 11:20:47 on dist-test-slave-srst"
server_key: "57d4e7a7ed36bfde184a759ed4d5289c"
server_key_iv: "5c48b53a4352aa07833f783dccf15a1a"
server_key_version: "encryptionkey@0"
I20260929 11:20:47.144223 3184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:20:47.147639 3199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:47.149065 3184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:47.149233 3184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "ae600dff18f24370bb28ea8d0a681c12"
format_stamp: "Formatted at 2026-09-29 11:20:47 on dist-test-slave-srst"
server_key: "57d4e7a7ed36bfde184a759ed4d5289c"
server_key_iv: "5c48b53a4352aa07833f783dccf15a1a"
server_key_version: "encryptionkey@0"
I20260929 11:20:47.149438 3184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:47.164352 3184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:47.168183 3184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:47.168452 3184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:47.169416 3184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:47.170709 3184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:47.170817 3184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:47.170958 3184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:47.171028 3184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:47.202595 3184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32927
I20260929 11:20:47.202677 3311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32927 every 8 connection(s)
I20260929 11:20:47.203917 3184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:47.213506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3184
I20260929 11:20:47.213703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:47.214234 388 external_mini_cluster.cc:1428] Setting key 7dfecd8dc71c95f432605fb4feff02b6
Sep 29 11:20:47 dist-test-slave-srst krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:47.223846 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:47.206528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34637 (local address 127.0.97.62:44823)
0929 11:20:47.206727 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:47.206731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:47.207923 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 11:20:47.209747 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:47.209753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:47.209833 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:47.209942 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:47.211383 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:47.212136 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:47.213899 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:47.214532 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:47.214672 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:47.218481 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:47.218487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:47.218490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:47.218521 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:47.221039 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:47.221884 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:47.221888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:47.221890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:47.221996 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:47.222352 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:47.222355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:47.222357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:47.222664 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:47.222777 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:47.223436 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 11:20:47.223589 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:20:47.224821 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:47.207022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44823 (local address 127.0.97.3:34637)
0929 11:20:47.207755 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:47.207794 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:47.209481 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:47.210164 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:47.210172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:47.210688 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:47.211222 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:47.211238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:47.212288 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:47.212293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:47.213726 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:47.213737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:47.214764 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:47.214771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:47.215077 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:47.215852 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:47.215878 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:47.218286 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:47.221168 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:47.221175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:47.221187 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:47.221707 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:47.222082 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:47.222085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:47.222088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:47.222257 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:47.222766 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:47.222793 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:47.223094 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 11:20:47.223326 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 11:20:47.223681 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260929 11:20:47.226152 3312 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44823
I20260929 11:20:47.226428 3312 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:47.227039 3312 heartbeater.cc:511] Master 127.0.97.62:44823 requested a full tablet report, sending...
I20260929 11:20:47.228421 2855 ts_manager.cc:194] Registered new tserver with Master: ae600dff18f24370bb28ea8d0a681c12 (127.0.97.3:32927)
I20260929 11:20:47.229404 2855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34637
I20260929 11:20:47.229486 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:20:47.262053 3047 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:20:47.259746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.0.97.62:44823)
0929 11:20:47.259959 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:47.259964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:47.260451 (+ 487us) server_negotiation.cc:407] Connection header received
0929 11:20:47.260643 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:47.260647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:47.260708 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:47.260803 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:47.261867 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53352: BlockingRecv error: recv got EOF from 127.0.0.1:53352 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:20:47.265187 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2917
W20260929 11:20:47.268014 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2917: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34527: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:20:47.268198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 2917
I20260929 11:20:47.283329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3051
W20260929 11:20:47.285905 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3051: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35781: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:20:47.286065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3051
I20260929 11:20:47.301100 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3184
W20260929 11:20:47.303752 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3184: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:32927: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:20:47.303906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3184
I20260929 11:20:47.318933 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 2824
I20260929 11:20:47.320015 2886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:47.426739 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 2824
2026-09-29T11:20:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3418 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:47 dist-test-slave-srst krb5kdc[3329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:47 dist-test-slave-srst krb5kdc[3329](info): set up 2 sockets
Sep 29 11:20:47 dist-test-slave-srst krb5kdc[3329](info): commencing operation
krb5kdc: starting...
W20260929 11:20:48.504943 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:48 dist-test-slave-srst krb5kdc[3329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:48.688112 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:49.016294 3342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:49.016630 3342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:49.016791 3342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:49.024542 3342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:49.024727 3342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:49.024844 3342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:49.024945 3342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:49.025041 3342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:49.035265 3342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:49.036695 3342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:49.038128 3342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:49.050092 3350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:49.050128 3347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:49.050149 3348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:49.053993 3342 server_base.cc:1034] running on GCE node
I20260929 11:20:49.054738 3342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:49.056026 3342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:49.057219 3342 hybrid_clock.cc:648] HybridClock initialized: now 1790680849057172 us; error 37 us; skew 500 ppm
I20260929 11:20:49.057531 3342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:49 dist-test-slave-srst krb5kdc[3329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:49.063748 3342 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:49.065301 3342 webserver.cc:460] Webserver started at http://127.0.97.62:33005/ using document root <none> and password file <none>
I20260929 11:20:49.065991 3342 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:49.066138 3342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:49.066370 3342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:49.069273 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a"
format_stamp: "Formatted at 2026-09-29 11:20:49 on dist-test-slave-srst"
server_key: "bc6e11ea7b55b210f29e9e3f8bc81276"
server_key_iv: "27c2823a1a19bcf57ecd7d2221ca0c45"
server_key_version: "encryptionkey@0"
I20260929 11:20:49.069991 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a"
format_stamp: "Formatted at 2026-09-29 11:20:49 on dist-test-slave-srst"
server_key: "bc6e11ea7b55b210f29e9e3f8bc81276"
server_key_iv: "27c2823a1a19bcf57ecd7d2221ca0c45"
server_key_version: "encryptionkey@0"
I20260929 11:20:49.074858 3342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:20:49.078457 3357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:49.079725 3342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:20:49.079891 3342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a"
format_stamp: "Formatted at 2026-09-29 11:20:49 on dist-test-slave-srst"
server_key: "bc6e11ea7b55b210f29e9e3f8bc81276"
server_key_iv: "27c2823a1a19bcf57ecd7d2221ca0c45"
server_key_version: "encryptionkey@0"
I20260929 11:20:49.080060 3342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:49.090551 3342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:49.094278 3342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:49.094551 3342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:49.112177 3342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43585
I20260929 11:20:49.112208 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43585 every 8 connection(s)
I20260929 11:20:49.113528 3342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:49.114739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3342
I20260929 11:20:49.114936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:49.115731 388 external_mini_cluster.cc:1428] Setting key 96443bc0517f983ad8b4b415a1e2385c
I20260929 11:20:49.118338 3409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:20:49 dist-test-slave-srst krb5kdc[3329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:49.129591 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a: Bootstrap starting.
I20260929 11:20:49.132860 3409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:49.134112 3409 log.cc:824] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:49.137107 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:49.117629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.0.97.62:43585)
0929 11:20:49.118425 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:49.118444 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:49.118493 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:20:49.120051 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:49.120085 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:49.120477 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:49.120935 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:49.122355 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:49.123692 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:49.124949 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:49.125658 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:49.125853 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:49.130076 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:49.130083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:49.130098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:49.130131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:49.133107 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:49.133842 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:49.133848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:49.133853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:49.134039 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:49.134410 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:49.134414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:49.134416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:49.135138 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:49.135465 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:49.135733 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 11:20:49.136083 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":365,"threads_started":1}
I20260929 11:20:49.137254 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a: No bootstrap required, opened a new log
I20260929 11:20:49.140684 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:20:49.147365 3409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260929 11:20:49.147804 3409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:49.147912 3409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100b6cbd1d4c44edbdcbcaee2c0b340a, State: Initialized, Role: FOLLOWER
I20260929 11:20:49.148476 3409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260929 11:20:49.148803 3409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:49.148943 3409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:49.149117 3409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:49.151177 3409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260929 11:20:49.151667 3409 leader_election.cc:302] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100b6cbd1d4c44edbdcbcaee2c0b340a; no voters:
I20260929 11:20:49.152350 3409 leader_election.cc:288] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:49.152448 3415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:49.154877 3409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:49.156299 3415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [term 1 LEADER]: Becoming Leader. State: Replica: 100b6cbd1d4c44edbdcbcaee2c0b340a, State: Running, Role: LEADER
I20260929 11:20:49.157054 3415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } }
I20260929 11:20:49.160125 3417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100b6cbd1d4c44edbdcbcaee2c0b340a. Latest consensus state: current_term: 1 leader_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } }
I20260929 11:20:49.160539 3417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:49.161237 3416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100b6cbd1d4c44edbdcbcaee2c0b340a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43585 } } }
I20260929 11:20:49.161554 3416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:49.173676 3426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:49.178613 3426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:49.186903 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:49.178935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.0.97.62:43585)
0929 11:20:49.179104 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:49.179111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:49.179641 (+ 530us) server_negotiation.cc:407] Connection header received
0929 11:20:49.179836 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:49.179842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:49.179915 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:49.180028 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:49.181013 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:49.181781 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:49.183280 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:49.183953 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:49.184122 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:49.185267 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:49.185273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:49.185278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:20:49.186034 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:49.186091 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:20:49.186129 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:49.186234 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:49.186421 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 11:20:49.186508 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 11:20:49.199231 3426 catalog_manager.cc:1347] Generated new cluster ID: 43ea43c2520648d6a40ad372aa2f04af
I20260929 11:20:49.199385 3426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:49.226131 3426 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:49.228068 3426 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:49.242980 3426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100b6cbd1d4c44edbdcbcaee2c0b340a: Generated new TSK 0
I20260929 11:20:49.244128 3426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:20:49.252511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3342
I20260929 11:20:49.253638 3404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:49.345816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3342
2026-09-29T11:20:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1914 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:49 dist-test-slave-srst krb5kdc[3444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:49 dist-test-slave-srst krb5kdc[3444](info): set up 2 sockets
Sep 29 11:20:49 dist-test-slave-srst krb5kdc[3444](info): commencing operation
krb5kdc: starting...
W20260929 11:20:50.412953 388 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:50 dist-test-slave-srst krb5kdc[3444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:50.595677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40009
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:50.902909 3457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:50.903288 3457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:50.903476 3457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:50.910821 3457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:50.910960 3457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:50.911043 3457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:50.911125 3457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:50.911195 3457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:50.921459 3457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:50.922847 3457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:50.924198 3457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:50.935702 3463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:50.935806 3462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:50.936002 3465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:50.936842 3457 server_base.cc:1034] running on GCE node
I20260929 11:20:50.937525 3457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:50.938833 3457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:50.940021 3457 hybrid_clock.cc:648] HybridClock initialized: now 1790680850939985 us; error 28 us; skew 500 ppm
I20260929 11:20:50.940353 3457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:50 dist-test-slave-srst krb5kdc[3444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:50.943419 3457 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:50.944931 3457 webserver.cc:460] Webserver started at http://127.0.97.62:34565/ using document root <none> and password file <none>
I20260929 11:20:50.945586 3457 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:50.945729 3457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:50.946019 3457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:50.948920 3457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "baa98a68087248b5ad04336b98cabbe4"
format_stamp: "Formatted at 2026-09-29 11:20:50 on dist-test-slave-srst"
server_key: "efee24fb1ccc3e45b2493d1075754b35"
server_key_iv: "e7cb3188a58748628aaec32cca555481"
server_key_version: "encryptionkey@0"
I20260929 11:20:50.949667 3457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "baa98a68087248b5ad04336b98cabbe4"
format_stamp: "Formatted at 2026-09-29 11:20:50 on dist-test-slave-srst"
server_key: "efee24fb1ccc3e45b2493d1075754b35"
server_key_iv: "e7cb3188a58748628aaec32cca555481"
server_key_version: "encryptionkey@0"
I20260929 11:20:50.954512 3457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 11:20:50.957643 3472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:50.958822 3457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 11:20:50.959020 3457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "baa98a68087248b5ad04336b98cabbe4"
format_stamp: "Formatted at 2026-09-29 11:20:50 on dist-test-slave-srst"
server_key: "efee24fb1ccc3e45b2493d1075754b35"
server_key_iv: "e7cb3188a58748628aaec32cca555481"
server_key_version: "encryptionkey@0"
I20260929 11:20:50.959203 3457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:50.989159 3457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:50.993507 3457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:50.993789 3457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:51.011547 3457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40009
I20260929 11:20:51.011584 3523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40009 every 8 connection(s)
I20260929 11:20:51.012887 3457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:51.016454 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:51.021747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3457
I20260929 11:20:51.021953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:51.022753 388 external_mini_cluster.cc:1428] Setting key c5c40ed136e6146f9863173a5f5f611f
I20260929 11:20:51.027784 3524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4: Bootstrap starting.
I20260929 11:20:51.031447 3524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:51.032702 3524 log.cc:824] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:51.036079 3524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4: No bootstrap required, opened a new log
Sep 29 11:20:51 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:51.044090 3524 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa98a68087248b5ad04336b98cabbe4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260929 11:20:51.044436 3524 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:51.044548 3524 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa98a68087248b5ad04336b98cabbe4, State: Initialized, Role: FOLLOWER
I20260929 11:20:51.045096 3524 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa98a68087248b5ad04336b98cabbe4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260929 11:20:51.045362 3524 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:51.045503 3524 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:51.045676 3524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:51.046049 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:51.024933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.97.62:40009)
0929 11:20:51.025820 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:51.025831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:51.025872 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:20:51.027491 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:51.027519 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:51.027978 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:51.028318 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:51.030062 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:51.031314 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:51.033045 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:51.034111 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:51.034240 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:51.038708 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:51.038717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:51.038733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:51.038772 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:51.041826 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:51.042479 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:51.042485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:51.042491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:51.042638 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:51.042998 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:51.043002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:51.043004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:51.043552 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:51.043845 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:51.044223 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 11:20:51.044840 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":444,"threads_started":1}
I20260929 11:20:51.047891 3524 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa98a68087248b5ad04336b98cabbe4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260929 11:20:51.048352 3524 leader_election.cc:302] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa98a68087248b5ad04336b98cabbe4; no voters:
I20260929 11:20:51.048982 3524 leader_election.cc:288] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:51.049094 3529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:51.049727 3529 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [term 1 LEADER]: Becoming Leader. State: Replica: baa98a68087248b5ad04336b98cabbe4, State: Running, Role: LEADER
I20260929 11:20:51.050262 3529 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa98a68087248b5ad04336b98cabbe4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260929 11:20:51.051100 3524 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:51.052986 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa98a68087248b5ad04336b98cabbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa98a68087248b5ad04336b98cabbe4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } } }
I20260929 11:20:51.053310 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:51.055780 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa98a68087248b5ad04336b98cabbe4. Latest consensus state: current_term: 1 leader_uuid: "baa98a68087248b5ad04336b98cabbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa98a68087248b5ad04336b98cabbe4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } } }
I20260929 11:20:51.056027 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:51.057814 3534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:51.062668 3534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:51.070838 3534 catalog_manager.cc:1347] Generated new cluster ID: 64baef1c007b4266b80ae51022ef74cf
I20260929 11:20:51.070935 3534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:51.090601 3534 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:51.091967 3534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:51.105369 3534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa98a68087248b5ad04336b98cabbe4: Generated new TSK 0
I20260929 11:20:51.106242 3534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:51.157145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:51.477387 3550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:51.477699 3550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:51.477876 3550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:51.485077 3550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:51.485198 3550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:51.485368 3550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:51.494863 3550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:51.496058 3550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:51.497308 3550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:51.510011 3555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:51.510140 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:51.510075 3558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:51.511092 3550 server_base.cc:1034] running on GCE node
I20260929 11:20:51.511668 3550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:51.512624 3550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:51.513841 3550 hybrid_clock.cc:648] HybridClock initialized: now 1790680851513768 us; error 70 us; skew 500 ppm
I20260929 11:20:51.514144 3550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:51 dist-test-slave-srst krb5kdc[3444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:51.517315 3550 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:51.519060 3550 webserver.cc:460] Webserver started at http://127.0.97.1:42085/ using document root <none> and password file <none>
I20260929 11:20:51.519822 3550 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:51.519965 3550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:51.520264 3550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:51.523151 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "fcebb074b7d34ab18bdd0faaddc0133b"
format_stamp: "Formatted at 2026-09-29 11:20:51 on dist-test-slave-srst"
server_key: "77c56eff5b87a4f15dda96e0fed71fa3"
server_key_iv: "3c4eaed640ca43223b4e22135fba4a4f"
server_key_version: "encryptionkey@0"
I20260929 11:20:51.523904 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fcebb074b7d34ab18bdd0faaddc0133b"
format_stamp: "Formatted at 2026-09-29 11:20:51 on dist-test-slave-srst"
server_key: "77c56eff5b87a4f15dda96e0fed71fa3"
server_key_iv: "3c4eaed640ca43223b4e22135fba4a4f"
server_key_version: "encryptionkey@0"
I20260929 11:20:51.528932 3550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:20:51.532153 3565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:51.533478 3550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:51.533636 3550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "fcebb074b7d34ab18bdd0faaddc0133b"
format_stamp: "Formatted at 2026-09-29 11:20:51 on dist-test-slave-srst"
server_key: "77c56eff5b87a4f15dda96e0fed71fa3"
server_key_iv: "3c4eaed640ca43223b4e22135fba4a4f"
server_key_version: "encryptionkey@0"
I20260929 11:20:51.533818 3550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:51.550123 3550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:51.554041 3550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:51.554312 3550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:51.555204 3550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:51.556658 3550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:51.556746 3550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:51.556910 3550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:51.556989 3550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:51.588510 3550 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36071
I20260929 11:20:51.588554 3677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36071 every 8 connection(s)
I20260929 11:20:51.589790 3550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:51.596503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3550
I20260929 11:20:51.596725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:51.597523 388 external_mini_cluster.cc:1428] Setting key 5def44d571ad8edb77f0bccad4fd3589
Sep 29 11:20:51 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:51.612409 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:51.592256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40365 (local address 127.0.97.62:40009)
0929 11:20:51.592740 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:51.592744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:51.593620 (+ 876us) server_negotiation.cc:407] Connection header received
0929 11:20:51.595382 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:51.595388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:51.595478 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:51.595592 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:51.597542 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:51.598478 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:51.600848 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:51.601248 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:51.601351 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:51.605825 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:51.605831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:51.605835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:51.605866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:51.609172 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:51.609868 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:51.609876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:51.609882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:51.610048 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:51.610434 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:51.610442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:51.610447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:51.610843 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:51.610988 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:51.611649 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 11:20:51.611917 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
I20260929 11:20:51.613027 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:51.592752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40009 (local address 127.0.97.1:40365)
0929 11:20:51.593466 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:51.593503 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:51.595115 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:51.595825 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:51.595833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:51.596522 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:51.597369 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:51.597395 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:51.598639 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:51.598645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:51.600717 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:51.600733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:51.601475 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:51.601484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:51.601810 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:51.602515 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:51.602549 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:51.605581 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:51.609300 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:51.609307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:51.609321 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:51.609735 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:51.610149 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:51.610152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:51.610155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:51.610321 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:51.610956 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:51.610962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:51.611269 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 11:20:51.611507 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 11:20:51.611887 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":263,"threads_started":1}
I20260929 11:20:51.614720 3678 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40009
I20260929 11:20:51.615067 3678 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:51.615967 3678 heartbeater.cc:511] Master 127.0.97.62:40009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:20:51.618135 3489 ts_manager.cc:194] Registered new tserver with Master: fcebb074b7d34ab18bdd0faaddc0133b (127.0.97.1:36071)
I20260929 11:20:51.619517 3489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:51.641494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:51.957376 3684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:51.957650 3684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:51.957741 3684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:51.964852 3684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:51.964951 3684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:51.965054 3684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:51.974118 3684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:51.975260 3684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:51.976490 3684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:51.989255 3689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:51.989280 3692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:51.989217 3690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:51.990418 3684 server_base.cc:1034] running on GCE node
I20260929 11:20:51.991042 3684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:51.991918 3684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:51.993136 3684 hybrid_clock.cc:648] HybridClock initialized: now 1790680851993090 us; error 33 us; skew 500 ppm
I20260929 11:20:51.993461 3684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:51 dist-test-slave-srst krb5kdc[3444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:51.996649 3684 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:51.999837 3684 webserver.cc:460] Webserver started at http://127.0.97.2:44341/ using document root <none> and password file <none>
I20260929 11:20:52.000674 3684 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:52.000818 3684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:52.001179 3684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:52.005491 3684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "ec82ba90f75b44448db18b2c79311ce1"
format_stamp: "Formatted at 2026-09-29 11:20:51 on dist-test-slave-srst"
server_key: "a747313686f5054481e756b2710512f0"
server_key_iv: "dc468591b5f56d0cfe001cee69f44f79"
server_key_version: "encryptionkey@0"
I20260929 11:20:52.006286 3684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ec82ba90f75b44448db18b2c79311ce1"
format_stamp: "Formatted at 2026-09-29 11:20:51 on dist-test-slave-srst"
server_key: "a747313686f5054481e756b2710512f0"
server_key_iv: "dc468591b5f56d0cfe001cee69f44f79"
server_key_version: "encryptionkey@0"
I20260929 11:20:52.011831 3684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 11:20:52.016180 3699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:52.017851 3684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:52.018006 3684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "ec82ba90f75b44448db18b2c79311ce1"
format_stamp: "Formatted at 2026-09-29 11:20:51 on dist-test-slave-srst"
server_key: "a747313686f5054481e756b2710512f0"
server_key_iv: "dc468591b5f56d0cfe001cee69f44f79"
server_key_version: "encryptionkey@0"
I20260929 11:20:52.018205 3684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:52.053791 3684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:52.057947 3684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:52.058279 3684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:52.059278 3684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:52.060577 3684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:52.060667 3684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:52.060828 3684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:52.060881 3684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:52.093076 3684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33865
I20260929 11:20:52.093112 3811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33865 every 8 connection(s)
I20260929 11:20:52.094349 3684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:52.099225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3684
I20260929 11:20:52.099398 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:52.099965 388 external_mini_cluster.cc:1428] Setting key 8d6d1b1cacdf2f6eabcd7c985b2f38da
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:52.117321 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.096906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55907 (local address 127.0.97.62:40009)
0929 11:20:52.097076 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:52.097081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:52.098255 (+ 1174us) server_negotiation.cc:407] Connection header received
0929 11:20:52.100690 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:52.100696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:52.100802 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:52.100927 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:52.102631 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.103325 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.105881 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.106268 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.106367 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.110195 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:52.110204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:52.110212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:52.110256 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:52.113933 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.114649 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.114657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.114663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.114828 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.115280 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.115287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.115294 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.115682 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:52.115836 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:52.116571 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 11:20:52.116860 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:20:52.118434 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.097424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40009 (local address 127.0.97.2:55907)
0929 11:20:52.098094 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:52.098136 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:52.100335 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:52.101170 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:52.101178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:52.101686 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:52.102447 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:52.102472 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.103487 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.103496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:52.105720 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:52.105737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.106495 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.106504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:52.106821 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.107593 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:52.107619 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:52.109977 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:52.114084 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:52.114091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:52.114104 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:52.114512 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:52.114964 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:52.114968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:52.114971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:52.115145 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:52.115820 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:52.115827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:52.116145 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 11:20:52.116424 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 11:20:52.116940 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:20:52.120003 3812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40009
I20260929 11:20:52.120278 3812 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:52.121049 3812 heartbeater.cc:511] Master 127.0.97.62:40009 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 11:20:52.122622 3489 ts_manager.cc:194] Registered new tserver with Master: ec82ba90f75b44448db18b2c79311ce1 (127.0.97.2:33865)
I20260929 11:20:52.123906 3489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:52.145565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:52.472147 3817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:52.472467 3817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:52.472692 3817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:52.479892 3817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:52.480059 3817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:52.480243 3817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:52.489965 3817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:52.491245 3817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:52.492637 3817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:52.505733 3825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:52.505721 3823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:52.506304 3822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:52.506517 3817 server_base.cc:1034] running on GCE node
I20260929 11:20:52.507117 3817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:52.508028 3817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:52.509209 3817 hybrid_clock.cc:648] HybridClock initialized: now 1790680852509163 us; error 40 us; skew 500 ppm
I20260929 11:20:52.509524 3817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:52.512445 3817 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:52.513792 3817 webserver.cc:460] Webserver started at http://127.0.97.3:39137/ using document root <none> and password file <none>
I20260929 11:20:52.514442 3817 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:52.514570 3817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:52.514843 3817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:52.517597 3817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "f3114b79e2394ae095b7dda079eddabc"
format_stamp: "Formatted at 2026-09-29 11:20:52 on dist-test-slave-srst"
server_key: "312fd16c0844b7c0d89b6c9f1790f2fd"
server_key_iv: "235cc955b0ecacecb57d39abc5218d51"
server_key_version: "encryptionkey@0"
I20260929 11:20:52.518292 3817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f3114b79e2394ae095b7dda079eddabc"
format_stamp: "Formatted at 2026-09-29 11:20:52 on dist-test-slave-srst"
server_key: "312fd16c0844b7c0d89b6c9f1790f2fd"
server_key_iv: "235cc955b0ecacecb57d39abc5218d51"
server_key_version: "encryptionkey@0"
I20260929 11:20:52.523218 3817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:20:52.526429 3832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:52.527683 3817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:52.527830 3817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "f3114b79e2394ae095b7dda079eddabc"
format_stamp: "Formatted at 2026-09-29 11:20:52 on dist-test-slave-srst"
server_key: "312fd16c0844b7c0d89b6c9f1790f2fd"
server_key_iv: "235cc955b0ecacecb57d39abc5218d51"
server_key_version: "encryptionkey@0"
I20260929 11:20:52.528025 3817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:52.542524 3817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:52.546371 3817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:52.546631 3817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:52.547572 3817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:52.548977 3817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:52.549083 3817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:52.549214 3817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:52.549283 3817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:52.580236 3817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46439
I20260929 11:20:52.580288 3944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46439 every 8 connection(s)
I20260929 11:20:52.581497 3817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:52.582703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3817
I20260929 11:20:52.582885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:52.583441 388 external_mini_cluster.cc:1428] Setting key 1b05fb46226e9deaf2b146b53dbad8d7
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:52.602695 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.584230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51205 (local address 127.0.97.62:40009)
0929 11:20:52.584401 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:52.584405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:52.585851 (+ 1446us) server_negotiation.cc:407] Connection header received
0929 11:20:52.587635 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:52.587639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:52.587713 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:52.587796 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:52.589743 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.590483 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.592733 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.593172 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.593365 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.597049 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:52.597054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:52.597058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:52.597090 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:52.600048 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.600734 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.600738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.600740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.600868 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.601267 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.601270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.601272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.601554 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:52.601688 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:52.602289 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 11:20:52.602446 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:20:52.603583 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.584908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40009 (local address 127.0.97.3:51205)
0929 11:20:52.585686 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:52.585736 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:52.587414 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:52.588051 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:52.588063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:52.588780 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:52.589577 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:52.589600 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.590638 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.590643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:52.592579 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:52.592617 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.593442 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.593449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:52.593730 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.594590 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:52.594620 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:52.596814 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:52.600166 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:52.600172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:52.600183 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:52.600622 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:52.600975 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:52.600977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:52.600980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:52.601140 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:52.601682 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:52.601688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:52.601980 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 11:20:52.602187 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 11:20:52.602527 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":274,"threads_started":1}
I20260929 11:20:52.605020 3945 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40009
I20260929 11:20:52.605290 3945 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:52.605989 3945 heartbeater.cc:511] Master 127.0.97.62:40009 requested a full tablet report, sending...
I20260929 11:20:52.607249 3489 ts_manager.cc:194] Registered new tserver with Master: f3114b79e2394ae095b7dda079eddabc (127.0.97.3:46439)
I20260929 11:20:52.608034 3489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51205
I20260929 11:20:52.610607 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:20:52.622313 3678 heartbeater.cc:503] Master 127.0.97.62:40009 was elected leader, sending a full tablet report...
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:52.656672 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.643403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.0.97.62:40009)
0929 11:20:52.643593 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:52.643597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:52.644044 (+ 447us) server_negotiation.cc:407] Connection header received
0929 11:20:52.644215 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:52.644219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:52.644279 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:52.644368 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:52.645651 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.646387 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.647545 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.647916 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.648015 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.651545 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:52.651549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:52.651552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:52.651598 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:52.654167 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.654679 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.654682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.654684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.654810 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.655247 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.655252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.655256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.655592 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:52.655830 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:52.656121 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:20:52.656321 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 11:20:52.657181 3489 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44066
I20260929 11:20:52.658967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3550
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:52.673501 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.659826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.0.97.1:36071)
0929 11:20:52.660370 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:52.660376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:52.660397 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:20:52.660446 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:52.660452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:52.660748 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:52.660895 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:52.661974 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.662975 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.664122 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.664726 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.664851 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.668386 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:52.668395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:52.668408 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:52.668440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:52.671140 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.671723 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.671729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.671731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.671846 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.672133 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.672136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.672138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.672433 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:52.672704 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:52.672918 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 11:20:52.673113 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":239,"threads_started":1}
W20260929 11:20:52.674755 3673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42894
W20260929 11:20:52.675807 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:52.676040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3550
I20260929 11:20:52.691684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3684
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:20:52.706422 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.692514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.0.97.2:33865)
0929 11:20:52.693057 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:52.693063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:52.693087 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:20:52.693146 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:52.693151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:52.693345 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:52.693519 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:52.694653 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.695596 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.696838 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.697355 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.697501 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.701207 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:52.701216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:52.701229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:52.701257 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:52.703737 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.704353 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.704359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.704361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.704484 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.704928 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.704937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.704940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.705235 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:52.705524 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:52.705814 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 11:20:52.706067 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
W20260929 11:20:52.707656 3807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51472
W20260929 11:20:52.708701 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:52.708853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3684
I20260929 11:20:52.728909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3817
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680852, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:20:52.748970 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:52.730024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.97.3:46439)
0929 11:20:52.730708 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:52.730715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:52.730770 (+ 55us) server_negotiation.cc:407] Connection header received
0929 11:20:52.730845 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:52.730854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:52.731169 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:52.731333 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:52.732794 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.734245 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.735778 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:52.736331 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:52.736463 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:52.741938 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:52.741948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:52.741962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:52.741999 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:52.745986 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.746586 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.746596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.746601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.746739 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:52.747179 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:52.747184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:52.747190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:52.747610 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:52.747910 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:52.748302 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 11:20:52.748539 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":381,"threads_started":1}
W20260929 11:20:52.750406 3940 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51818
W20260929 11:20:52.751852 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3817: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:20:52.752024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3817
I20260929 11:20:52.768396 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3457
I20260929 11:20:52.769551 3519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:52.868789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3457
2026-09-29T11:20:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3526 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3962](info): set up 2 sockets
Sep 29 11:20:52 dist-test-slave-srst krb5kdc[3962](info): commencing operation
krb5kdc: starting...
W20260929 11:20:53.933128 388 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:54 dist-test-slave-srst krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:54.120239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:54.434221 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:54.434561 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:54.434719 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:54.441779 3975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:54.441941 3975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:54.442050 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:54.442143 3975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:54.442222 3975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:54.451792 3975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:54.453106 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:54.454411 3975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:54.465888 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:54.465996 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:54.465932 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:54.466997 3975 server_base.cc:1034] running on GCE node
I20260929 11:20:54.467959 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:54.469466 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:54.470715 3975 hybrid_clock.cc:648] HybridClock initialized: now 1790680854470674 us; error 46 us; skew 500 ppm
I20260929 11:20:54.471068 3975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:54 dist-test-slave-srst krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:54.478554 3975 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:54.480126 3975 webserver.cc:460] Webserver started at http://127.0.97.62:33791/ using document root <none> and password file <none>
I20260929 11:20:54.480937 3975 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:54.481076 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:54.481393 3975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:54.484580 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "42f5c0f17d904db79535073bf2c49d3d"
format_stamp: "Formatted at 2026-09-29 11:20:54 on dist-test-slave-srst"
server_key: "902de6c54da02c8998f456d6f2bdcd5a"
server_key_iv: "bcbb7cb5eddd8ccd1c0f769d81f3f828"
server_key_version: "encryptionkey@0"
I20260929 11:20:54.485443 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "42f5c0f17d904db79535073bf2c49d3d"
format_stamp: "Formatted at 2026-09-29 11:20:54 on dist-test-slave-srst"
server_key: "902de6c54da02c8998f456d6f2bdcd5a"
server_key_iv: "bcbb7cb5eddd8ccd1c0f769d81f3f828"
server_key_version: "encryptionkey@0"
I20260929 11:20:54.490573 3975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:20:54.493950 3990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:54.495257 3975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:20:54.495436 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "42f5c0f17d904db79535073bf2c49d3d"
format_stamp: "Formatted at 2026-09-29 11:20:54 on dist-test-slave-srst"
server_key: "902de6c54da02c8998f456d6f2bdcd5a"
server_key_iv: "bcbb7cb5eddd8ccd1c0f769d81f3f828"
server_key_version: "encryptionkey@0"
I20260929 11:20:54.495628 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:54.532096 3975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:54.536041 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:54.536326 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:54.554209 3975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46805
I20260929 11:20:54.554247 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46805 every 8 connection(s)
I20260929 11:20:54.555496 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:54.557852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 3975
I20260929 11:20:54.558065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:54.558670 388 external_mini_cluster.cc:1428] Setting key ba07ccef678a06a3b2de7cfcd897e770
I20260929 11:20:54.559214 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:54.573647 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d: Bootstrap starting.
Sep 29 11:20:54 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:54.576730 4042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:54.577741 4042 log.cc:824] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:54.580297 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d: No bootstrap required, opened a new log
I20260929 11:20:54.581857 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:54.560551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.0.97.62:46805)
0929 11:20:54.561384 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:54.561395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:54.561437 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:20:54.563677 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:54.563699 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:54.564108 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:54.564439 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:54.566412 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:54.567772 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:54.569596 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:54.570319 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:54.570467 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:54.575025 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:54.575034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:54.575048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:54.575084 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:54.578109 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:54.578783 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:54.578789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:54.578795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:54.578970 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:54.579338 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:54.579343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:54.579345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:54.579812 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:54.580102 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:54.580498 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 11:20:54.580939 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":371,"threads_started":1}
I20260929 11:20:54.587802 4042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5c0f17d904db79535073bf2c49d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46805 } }
I20260929 11:20:54.588156 4042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:54.588263 4042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f5c0f17d904db79535073bf2c49d3d, State: Initialized, Role: FOLLOWER
I20260929 11:20:54.588868 4042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5c0f17d904db79535073bf2c49d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46805 } }
I20260929 11:20:54.589097 4042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:54.589238 4042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:54.589416 4042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:54.591804 4042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5c0f17d904db79535073bf2c49d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46805 } }
I20260929 11:20:54.592279 4042 leader_election.cc:302] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f5c0f17d904db79535073bf2c49d3d; no voters:
I20260929 11:20:54.592860 4042 leader_election.cc:288] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:54.592989 4047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:54.593662 4047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [term 1 LEADER]: Becoming Leader. State: Replica: 42f5c0f17d904db79535073bf2c49d3d, State: Running, Role: LEADER
I20260929 11:20:54.594233 4047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5c0f17d904db79535073bf2c49d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46805 } }
I20260929 11:20:54.594970 4042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:54.596973 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f5c0f17d904db79535073bf2c49d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5c0f17d904db79535073bf2c49d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46805 } } }
I20260929 11:20:54.597348 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:54.597096 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f5c0f17d904db79535073bf2c49d3d. Latest consensus state: current_term: 1 leader_uuid: "42f5c0f17d904db79535073bf2c49d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5c0f17d904db79535073bf2c49d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46805 } } }
I20260929 11:20:54.597472 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:54.599937 4052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:54.604219 4052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:54.612874 4052 catalog_manager.cc:1347] Generated new cluster ID: 6d1cc50f7fa24c199a94d9ceef5748bc
I20260929 11:20:54.613040 4052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:54.665661 4052 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:54.666985 4052 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:54.672186 4052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f5c0f17d904db79535073bf2c49d3d: Generated new TSK 0
I20260929 11:20:54.673089 4052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:54.730310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46805
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:55.052838 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:55.053090 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:55.053185 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:55.059973 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:55.060077 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:55.060182 4068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:55.069484 4068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:55.070664 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:55.071914 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:55.085879 4073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:55.085844 4076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:55.085909 4074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:55.086776 4068 server_base.cc:1034] running on GCE node
I20260929 11:20:55.087364 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:55.088276 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:55.089496 4068 hybrid_clock.cc:648] HybridClock initialized: now 1790680855089450 us; error 38 us; skew 500 ppm
I20260929 11:20:55.089824 4068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:55 dist-test-slave-srst krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:55.092794 4068 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:55.094126 4068 webserver.cc:460] Webserver started at http://127.0.97.1:42089/ using document root <none> and password file <none>
I20260929 11:20:55.094781 4068 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:55.094879 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:55.095124 4068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:55.097949 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "7a0abe1de20f45eaad00d915d0d14e33"
format_stamp: "Formatted at 2026-09-29 11:20:55 on dist-test-slave-srst"
server_key: "6de3f24ff47edec5f113adbfa40fce64"
server_key_iv: "a6ac3adc6b29f0c3ced03023cee00266"
server_key_version: "encryptionkey@0"
I20260929 11:20:55.098757 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7a0abe1de20f45eaad00d915d0d14e33"
format_stamp: "Formatted at 2026-09-29 11:20:55 on dist-test-slave-srst"
server_key: "6de3f24ff47edec5f113adbfa40fce64"
server_key_iv: "a6ac3adc6b29f0c3ced03023cee00266"
server_key_version: "encryptionkey@0"
I20260929 11:20:55.103758 4068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:20:55.106762 4083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:55.107997 4068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 11:20:55.108148 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "7a0abe1de20f45eaad00d915d0d14e33"
format_stamp: "Formatted at 2026-09-29 11:20:55 on dist-test-slave-srst"
server_key: "6de3f24ff47edec5f113adbfa40fce64"
server_key_iv: "a6ac3adc6b29f0c3ced03023cee00266"
server_key_version: "encryptionkey@0"
I20260929 11:20:55.108328 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:55.132138 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:55.135852 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:55.136125 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:55.137076 4068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:55.138350 4068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:55.138447 4068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:55.138588 4068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:55.138657 4068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:55.170029 4068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37173
I20260929 11:20:55.170109 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37173 every 8 connection(s)
I20260929 11:20:55.171263 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:55.178406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4068
I20260929 11:20:55.178617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:55.179248 388 external_mini_cluster.cc:1428] Setting key 47c9d865de54f4efdb3987958e25e44e
Sep 29 11:20:55 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:55.191061 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:55.173759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54815 (local address 127.0.97.62:46805)
0929 11:20:55.174118 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:55.174122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:55.175199 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 11:20:55.176998 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:55.177003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:55.177090 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:55.177201 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:55.178766 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.179534 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.181592 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.181998 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.182135 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:55.185596 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:55.185601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:55.185604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:55.185633 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:55.188102 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:55.188865 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:55.188870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:55.188877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:55.188986 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:55.189409 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:55.189414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:55.189422 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:55.189678 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:55.189821 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:55.190512 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 11:20:55.190719 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":181,"threads_started":1}
I20260929 11:20:55.192054 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:55.174342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46805 (local address 127.0.97.1:54815)
0929 11:20:55.175058 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:55.175094 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:55.176732 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:55.177426 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:55.177434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:55.177948 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:55.178577 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:55.178594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.179673 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.179678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:55.181434 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:55.181445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.182229 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.182235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:55.182474 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:55.183140 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:55.183163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:55.185375 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:55.188236 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:55.188244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:55.188262 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:55.188752 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:55.189087 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:55.189093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:55.189097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:55.189275 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:55.189815 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:55.189823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:55.190146 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 11:20:55.190405 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 11:20:55.190923 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":318,"threads_started":1}
I20260929 11:20:55.193521 4196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46805
I20260929 11:20:55.193812 4196 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:55.194475 4196 heartbeater.cc:511] Master 127.0.97.62:46805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:20:55.196663 4007 ts_manager.cc:194] Registered new tserver with Master: 7a0abe1de20f45eaad00d915d0d14e33 (127.0.97.1:37173)
I20260929 11:20:55.198153 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:55.220126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46805
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:55.535337 4202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:55.535610 4202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:55.535705 4202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:55.542876 4202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:55.542991 4202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:55.543119 4202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:55.552671 4202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:55.553951 4202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:55.555251 4202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:55.569659 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:55.569698 4210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:55.569729 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:55.570796 4202 server_base.cc:1034] running on GCE node
I20260929 11:20:55.571377 4202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:55.572307 4202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:55.573530 4202 hybrid_clock.cc:648] HybridClock initialized: now 1790680855573483 us; error 33 us; skew 500 ppm
I20260929 11:20:55.573861 4202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:55 dist-test-slave-srst krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:55.576709 4202 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:55.578039 4202 webserver.cc:460] Webserver started at http://127.0.97.2:37119/ using document root <none> and password file <none>
I20260929 11:20:55.578665 4202 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:55.578795 4202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:55.579069 4202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:55.581677 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "fa75dd4816c9420babf3db00ae347d05"
format_stamp: "Formatted at 2026-09-29 11:20:55 on dist-test-slave-srst"
server_key: "6cbf904b84c266f76a629ad5a35cb8bd"
server_key_iv: "fe3bcddc6c20d7994e5776df94c90dca"
server_key_version: "encryptionkey@0"
I20260929 11:20:55.582424 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fa75dd4816c9420babf3db00ae347d05"
format_stamp: "Formatted at 2026-09-29 11:20:55 on dist-test-slave-srst"
server_key: "6cbf904b84c266f76a629ad5a35cb8bd"
server_key_iv: "fe3bcddc6c20d7994e5776df94c90dca"
server_key_version: "encryptionkey@0"
I20260929 11:20:55.587081 4202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:20:55.590147 4217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:55.591403 4202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:20:55.591564 4202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "fa75dd4816c9420babf3db00ae347d05"
format_stamp: "Formatted at 2026-09-29 11:20:55 on dist-test-slave-srst"
server_key: "6cbf904b84c266f76a629ad5a35cb8bd"
server_key_iv: "fe3bcddc6c20d7994e5776df94c90dca"
server_key_version: "encryptionkey@0"
I20260929 11:20:55.591758 4202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:55.608354 4202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:55.612368 4202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:55.612673 4202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:55.613533 4202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:55.614732 4202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:55.614799 4202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:55.614892 4202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:55.614935 4202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:55.645061 4202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35867
I20260929 11:20:55.645104 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35867 every 8 connection(s)
I20260929 11:20:55.646281 4202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:55.647032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4202
I20260929 11:20:55.647239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:55.647756 388 external_mini_cluster.cc:1428] Setting key 4695ba61aee84cdd4048b0ff89769297
Sep 29 11:20:55 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:20:55.666204 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:55.648845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52861 (local address 127.0.97.62:46805)
0929 11:20:55.649048 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:55.649054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:55.650551 (+ 1497us) server_negotiation.cc:407] Connection header received
0929 11:20:55.652305 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:55.652313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:55.652410 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:55.652530 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:55.653943 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.654807 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.656442 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.656887 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.657021 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:55.660543 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:55.660548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:55.660551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:55.660580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:55.663372 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:55.664045 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:55.664050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:55.664054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:55.664178 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:55.664578 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:55.664584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:55.664606 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:55.664865 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:55.665005 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:55.665614 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 11:20:55.665844 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:20:55.667171 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:55.649458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46805 (local address 127.0.97.2:52861)
0929 11:20:55.650378 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:55.650429 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:55.652041 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:55.652750 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:55.652758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:55.653245 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:55.653783 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:55.653799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.654962 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.654966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:55.656326 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:55.656337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:55.657109 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:55.657115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:55.657326 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:55.657970 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:55.657995 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:55.660322 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:55.663501 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:55.663510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:55.663527 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:55.663923 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:55.664281 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:55.664287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:55.664290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:55.664473 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:55.664979 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:55.664986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:55.665300 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 11:20:55.665551 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 11:20:55.666061 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":401,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 11:20:55.668535 4330 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46805
I20260929 11:20:55.668864 4330 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:55.669584 4330 heartbeater.cc:511] Master 127.0.97.62:46805 requested a full tablet report, sending...
I20260929 11:20:55.671043 4007 ts_manager.cc:194] Registered new tserver with Master: fa75dd4816c9420babf3db00ae347d05 (127.0.97.2:35867)
I20260929 11:20:55.671835 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:55.691452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46805
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:56.004136 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:56.004521 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:56.004824 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:56.013948 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:56.014066 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:56.014242 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:56.024235 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:56.025527 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:56.026868 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:56.039234 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:56.039327 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:56.039319 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:56.040323 4335 server_base.cc:1034] running on GCE node
I20260929 11:20:56.040930 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:56.041867 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:56.043061 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790680856043021 us; error 35 us; skew 500 ppm
I20260929 11:20:56.043350 4335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:56.046280 4335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:56.047631 4335 webserver.cc:460] Webserver started at http://127.0.97.3:38357/ using document root <none> and password file <none>
I20260929 11:20:56.048348 4335 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:56.048496 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:56.048827 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:56.051681 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "0f649175677441deb522b6cc29e3e2f9"
format_stamp: "Formatted at 2026-09-29 11:20:56 on dist-test-slave-srst"
server_key: "624a80bb8b9483bea2d8b61b515c62bb"
server_key_iv: "6bbc1fb079b61f84322cd1b3ef1a4571"
server_key_version: "encryptionkey@0"
I20260929 11:20:56.052459 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0f649175677441deb522b6cc29e3e2f9"
format_stamp: "Formatted at 2026-09-29 11:20:56 on dist-test-slave-srst"
server_key: "624a80bb8b9483bea2d8b61b515c62bb"
server_key_iv: "6bbc1fb079b61f84322cd1b3ef1a4571"
server_key_version: "encryptionkey@0"
I20260929 11:20:56.057530 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:20:56.061116 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:56.062358 4335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:20:56.062526 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "0f649175677441deb522b6cc29e3e2f9"
format_stamp: "Formatted at 2026-09-29 11:20:56 on dist-test-slave-srst"
server_key: "624a80bb8b9483bea2d8b61b515c62bb"
server_key_iv: "6bbc1fb079b61f84322cd1b3ef1a4571"
server_key_version: "encryptionkey@0"
I20260929 11:20:56.062716 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:56.094520 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:56.099017 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:56.099349 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:56.100332 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:56.101645 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:56.101712 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:56.101796 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:56.101838 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:56.132439 4335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35611
I20260929 11:20:56.132493 4462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35611 every 8 connection(s)
I20260929 11:20:56.133796 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:56.139571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4335
I20260929 11:20:56.139753 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:56.140280 388 external_mini_cluster.cc:1428] Setting key 4860aa91a1bea99488f29c317b764891
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:56.152257 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.136277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33609 (local address 127.0.97.62:46805)
0929 11:20:56.136473 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:56.136478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:56.137688 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 11:20:56.139369 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:56.139373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:56.139439 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:56.139509 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:56.141279 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.141998 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.143491 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.143879 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.144020 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.147022 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:56.147032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:56.147036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:56.147066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:56.149411 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.150106 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.150111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.150120 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.150227 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.150631 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.150636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.150639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.150904 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:56.151053 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:56.151711 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 11:20:56.151909 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:20:56.153214 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.136895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46805 (local address 127.0.97.3:33609)
0929 11:20:56.137549 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:56.137589 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:56.139083 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:56.139767 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:56.139780 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:56.140449 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:56.141123 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:56.141139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.142134 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.142139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:56.143331 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:56.143342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.144113 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.144119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:56.144332 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.144917 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:56.144941 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:56.146837 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:56.149576 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:56.149594 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:56.149609 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:56.149988 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:56.150338 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:56.150342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:56.150346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:56.150518 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:56.151036 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:56.151044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:56.151372 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 11:20:56.151629 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 11:20:56.152089 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":268,"threads_started":1}
I20260929 11:20:56.154692 4463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46805
I20260929 11:20:56.154978 4463 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:56.155613 4463 heartbeater.cc:511] Master 127.0.97.62:46805 requested a full tablet report, sending...
I20260929 11:20:56.156806 4007 ts_manager.cc:194] Registered new tserver with Master: 0f649175677441deb522b6cc29e3e2f9 (127.0.97.3:35611)
I20260929 11:20:56.157569 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33609
I20260929 11:20:56.166795 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 11:20:56.201105 4196 heartbeater.cc:503] Master 127.0.97.62:46805 was elected leader, sending a full tablet report...
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:20:56.213965 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.199670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.0.97.1:37173)
0929 11:20:56.200318 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:56.200325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:56.200350 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:20:56.200405 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:56.200411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:56.200662 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:56.200796 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:56.202385 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.203453 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.204555 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.205110 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.205195 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.208556 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:56.208566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:56.208580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:56.208673 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:56.211466 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.212104 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.212110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.212112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.212228 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.212627 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.212631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.212634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.212960 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:56.213162 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:56.213407 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 11:20:56.213585 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":322,"threads_started":1}
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:56.229811 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.217801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.0.97.62:46805)
0929 11:20:56.218009 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:56.218017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:56.218227 (+ 210us) server_negotiation.cc:407] Connection header received
0929 11:20:56.218404 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:56.218408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:56.218476 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:56.218573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:56.219656 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.220348 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.221612 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.221994 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.222121 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.225617 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:56.225621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:56.225624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:56.225654 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:56.227780 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.228291 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.228295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.228297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.228402 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.228746 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.228749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.228751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.229030 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:56.229215 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:56.229403 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 11:20:56.229578 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 11:20:56.230316 4007 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44304
I20260929 11:20:56.231230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4068
I20260929 11:20:56.242408 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.232014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.97.1:37173)
0929 11:20:56.232176 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:56.232180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:56.232499 (+ 319us) server_negotiation.cc:407] Connection header received
0929 11:20:56.232664 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:56.232673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:56.232755 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:56.232834 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:56.233888 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.234529 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.235666 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.236046 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.236134 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.238186 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:56.238191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:56.238194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:56.238220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:56.240464 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.241044 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.241047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.241049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.241153 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.241425 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.241428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.241429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.241649 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:56.241817 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:56.242017 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 11:20:56.242186 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:20:56.242940 4191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:56.321362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4068
I20260929 11:20:56.337495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4202
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:20:56.352806 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.338360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.0.97.2:35867)
0929 11:20:56.338894 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:56.338901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:56.338945 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:20:56.339011 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:56.339017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:56.339225 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:56.339352 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:56.340551 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.341548 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.342796 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.343343 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.343441 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.347452 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:56.347462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:56.347476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:56.347505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:56.350263 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.350967 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.350973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.350975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.351095 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.351418 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.351421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.351423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.351728 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:56.351976 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:56.352290 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 11:20:56.352471 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":309,"threads_started":1}
I20260929 11:20:56.354053 4325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:56.435195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4202
I20260929 11:20:56.450593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4335
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:20:56.465937 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:56.451423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.97.3:35611)
0929 11:20:56.452039 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:56.452046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:56.452073 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:20:56.452134 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:56.452140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:56.452355 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:56.452485 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:56.453678 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.454653 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.455801 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:56.456318 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:56.456408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:56.460706 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:56.460718 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:56.460732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:56.460761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:56.463426 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.464107 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.464114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.464116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.464226 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:56.464529 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:56.464532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:56.464534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:56.465011 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:56.465237 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:56.465468 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 11:20:56.465673 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":306,"threads_started":1}
I20260929 11:20:56.467490 4458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:56.545936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4335
I20260929 11:20:56.561520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 3975
I20260929 11:20:56.562625 4037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:20:56.661459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 3975
2026-09-29T11:20:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3793 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[4482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[4482](info): set up 2 sockets
Sep 29 11:20:56 dist-test-slave-srst krb5kdc[4482](info): commencing operation
krb5kdc: starting...
W20260929 11:20:57.737210 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.006s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:20:57 dist-test-slave-srst krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:20:57.928915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44373
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:58.241701 4495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:58.241987 4495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:58.242180 4495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:58.249320 4495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:20:58.249449 4495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:58.249537 4495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:58.249641 4495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:20:58.249708 4495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:20:58.259280 4495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:58.260519 4495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:58.261804 4495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:58.272433 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:58.272485 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:58.272626 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:58.273314 4495 server_base.cc:1034] running on GCE node
I20260929 11:20:58.274061 4495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:58.275357 4495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:58.276618 4495 hybrid_clock.cc:648] HybridClock initialized: now 1790680858276550 us; error 49 us; skew 500 ppm
I20260929 11:20:58.276959 4495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:58 dist-test-slave-srst krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:58.279893 4495 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:20:58.281065 4495 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:58.281198 4495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:58.281513 4495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:58.284385 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "08d5895e0a2b423bbe98d3f3f1817ad3"
format_stamp: "Formatted at 2026-09-29 11:20:58 on dist-test-slave-srst"
server_key: "73e83c88ba174a9b40bdaddfbd8d9881"
server_key_iv: "1bc539c98633d8c1d8bb493cc2663ff9"
server_key_version: "encryptionkey@0"
I20260929 11:20:58.285287 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "08d5895e0a2b423bbe98d3f3f1817ad3"
format_stamp: "Formatted at 2026-09-29 11:20:58 on dist-test-slave-srst"
server_key: "73e83c88ba174a9b40bdaddfbd8d9881"
server_key_iv: "1bc539c98633d8c1d8bb493cc2663ff9"
server_key_version: "encryptionkey@0"
I20260929 11:20:58.290467 4495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:20:58.293839 4509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:58.295142 4495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:20:58.295331 4495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "08d5895e0a2b423bbe98d3f3f1817ad3"
format_stamp: "Formatted at 2026-09-29 11:20:58 on dist-test-slave-srst"
server_key: "73e83c88ba174a9b40bdaddfbd8d9881"
server_key_iv: "1bc539c98633d8c1d8bb493cc2663ff9"
server_key_version: "encryptionkey@0"
I20260929 11:20:58.295538 4495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:58.312711 4495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:58.316462 4495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:58.316748 4495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:58.334993 4495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44373
I20260929 11:20:58.335013 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44373 every 8 connection(s)
I20260929 11:20:58.335927 4495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:20:58.339547 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:58.346293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4495
I20260929 11:20:58.346516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:20:58.347100 388 external_mini_cluster.cc:1428] Setting key 59c216a2903d60b16a9787f597a7b2ab
I20260929 11:20:58.349460 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3: Bootstrap starting.
I20260929 11:20:58.353381 4561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3: Neither blocks nor log segments found. Creating new log.
I20260929 11:20:58.354439 4561 log.cc:824] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3: Log is configured to *not* fsync() on all Append() calls
I20260929 11:20:58.357151 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3: No bootstrap required, opened a new log
Sep 29 11:20:58 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:58.366456 4561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44373 } }
I20260929 11:20:58.366876 4561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:20:58.366969 4561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d5895e0a2b423bbe98d3f3f1817ad3, State: Initialized, Role: FOLLOWER
I20260929 11:20:58.367666 4561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44373 } }
I20260929 11:20:58.367931 4561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:20:58.368058 4561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:20:58.368233 4561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:20:58.371148 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:58.348935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.0.97.62:44373)
0929 11:20:58.349712 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:58.349728 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:58.349774 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:20:58.352157 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:58.352183 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:58.352658 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:58.353085 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:58.354463 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:58.356217 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:58.357560 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:58.358532 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:58.358711 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:58.362474 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:58.362484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:58.362501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:58.362553 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:58.366227 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:58.367124 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:58.367131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:58.367138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:58.367355 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:58.367809 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:58.367814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:58.367817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:58.368513 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:58.368868 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:58.369262 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 11:20:58.369712 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":315,"threads_started":1}
I20260929 11:20:58.371471 4561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44373 } }
I20260929 11:20:58.372005 4561 leader_election.cc:302] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08d5895e0a2b423bbe98d3f3f1817ad3; no voters:
I20260929 11:20:58.372712 4561 leader_election.cc:288] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:20:58.372836 4566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:20:58.374833 4561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:20:58.379737 4566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 08d5895e0a2b423bbe98d3f3f1817ad3, State: Running, Role: LEADER
I20260929 11:20:58.380195 4566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44373 } }
I20260929 11:20:58.383344 4567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44373 } } }
I20260929 11:20:58.383365 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08d5895e0a2b423bbe98d3f3f1817ad3. Latest consensus state: current_term: 1 leader_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d5895e0a2b423bbe98d3f3f1817ad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44373 } } }
I20260929 11:20:58.383767 4567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:58.383765 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3 [sys.catalog]: This master's current role is: LEADER
I20260929 11:20:58.391041 4579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:20:58.396406 4579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:20:58.403790 4579 catalog_manager.cc:1347] Generated new cluster ID: da75b85a598e492d8fa13a117cfe59ed
I20260929 11:20:58.403903 4579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:20:58.415278 4579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:20:58.417047 4579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:20:58.422472 4579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3: Generated new TSK 0
I20260929 11:20:58.423245 4579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:20:58.468529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44373
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:58.792763 4587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:58.793195 4587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:58.793385 4587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:58.800575 4587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:58.800740 4587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:58.800865 4587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:20:58.810534 4587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:58.811731 4587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:58.813060 4587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:58.825987 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:58.826033 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:58.826084 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:58.826730 4587 server_base.cc:1034] running on GCE node
I20260929 11:20:58.827361 4587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:58.828284 4587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:58.829516 4587 hybrid_clock.cc:648] HybridClock initialized: now 1790680858829465 us; error 55 us; skew 500 ppm
I20260929 11:20:58.829846 4587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:58 dist-test-slave-srst krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:58.833062 4587 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:20:58.834233 4587 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:58.834329 4587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:58.834650 4587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:58.837705 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "dc716d7ec3474ef09d07cea3585a7674"
format_stamp: "Formatted at 2026-09-29 11:20:58 on dist-test-slave-srst"
server_key: "1ecaa3f50c03bd2ba7ec63a82ce2f21e"
server_key_iv: "a7f4abab86dc19d47d377089ac7ce80d"
server_key_version: "encryptionkey@0"
I20260929 11:20:58.838519 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dc716d7ec3474ef09d07cea3585a7674"
format_stamp: "Formatted at 2026-09-29 11:20:58 on dist-test-slave-srst"
server_key: "1ecaa3f50c03bd2ba7ec63a82ce2f21e"
server_key_iv: "a7f4abab86dc19d47d377089ac7ce80d"
server_key_version: "encryptionkey@0"
I20260929 11:20:58.843786 4587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:20:58.847433 4601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:58.848878 4587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:58.849021 4587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "dc716d7ec3474ef09d07cea3585a7674"
format_stamp: "Formatted at 2026-09-29 11:20:58 on dist-test-slave-srst"
server_key: "1ecaa3f50c03bd2ba7ec63a82ce2f21e"
server_key_iv: "a7f4abab86dc19d47d377089ac7ce80d"
server_key_version: "encryptionkey@0"
I20260929 11:20:58.849216 4587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:58.864311 4587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:58.868317 4587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:58.868692 4587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:58.869652 4587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:58.871013 4587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:58.871155 4587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:58.871299 4587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:58.871392 4587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:58.904433 4587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35443
I20260929 11:20:58.904474 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35443 every 8 connection(s)
I20260929 11:20:58.905295 4587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:20:58.908129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4587
I20260929 11:20:58.908342 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:20:58.909204 388 external_mini_cluster.cc:1428] Setting key 34e089df262997018dc6498206c8d834
Sep 29 11:20:58 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:58.926365 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:58.907594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59453 (local address 127.0.97.62:44373)
0929 11:20:58.908016 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:58.908020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:58.909142 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 11:20:58.911131 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:58.911136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:58.911244 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:58.911357 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:58.912774 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:58.913934 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:58.915388 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:58.916086 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:58.916230 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:58.920263 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:58.920268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:58.920271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:58.920300 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:58.923076 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:58.923806 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:58.923810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:58.923813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:58.923921 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:58.924425 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:58.924439 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:58.924446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:58.924887 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:58.925056 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:58.925738 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 11:20:58.925962 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":240,"threads_started":1}
I20260929 11:20:58.927003 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:58.908206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44373 (local address 127.0.97.1:59453)
0929 11:20:58.908998 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:58.909034 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:58.910840 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:58.911539 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:58.911547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:58.912041 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:58.912564 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:58.912580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:58.914066 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:58.914070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:58.915253 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:58.915263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:58.916352 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:58.916361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:58.916680 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:58.917405 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:58.917442 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:58.920046 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:58.923216 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:58.923224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:58.923249 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:58.923663 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:58.924028 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:58.924034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:58.924038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:58.924264 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:58.925014 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:58.925027 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:58.925348 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 11:20:58.925567 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 11:20:58.925953 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":363,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:20:58.928460 4714 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44373
I20260929 11:20:58.928797 4714 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:58.929416 4714 heartbeater.cc:511] Master 127.0.97.62:44373 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 11:20:58.931533 4526 ts_manager.cc:194] Registered new tserver with Master: dc716d7ec3474ef09d07cea3585a7674 (127.0.97.1:35443)
I20260929 11:20:58.932988 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:20:58.952168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44373
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:59.260749 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:59.261015 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:59.261109 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:59.268160 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:59.268268 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:59.268390 4720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:20:59.278266 4720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:59.279490 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:59.280766 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:59.293041 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:59.293005 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:59.292964 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:59.293919 4720 server_base.cc:1034] running on GCE node
I20260929 11:20:59.294559 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:59.295454 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:59.296686 4720 hybrid_clock.cc:648] HybridClock initialized: now 1790680859296621 us; error 46 us; skew 500 ppm
I20260929 11:20:59.296978 4720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:59 dist-test-slave-srst krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:59.300043 4720 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:20:59.301129 4720 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:59.301257 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:59.301570 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:59.304286 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b977a7942034f8e9b3d0febe490414f"
format_stamp: "Formatted at 2026-09-29 11:20:59 on dist-test-slave-srst"
server_key: "03d52eb3dae625915c4641b0fb1da50e"
server_key_iv: "f95677dfa76c3964dad66fc82a4e31c4"
server_key_version: "encryptionkey@0"
I20260929 11:20:59.305171 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b977a7942034f8e9b3d0febe490414f"
format_stamp: "Formatted at 2026-09-29 11:20:59 on dist-test-slave-srst"
server_key: "03d52eb3dae625915c4641b0fb1da50e"
server_key_iv: "f95677dfa76c3964dad66fc82a4e31c4"
server_key_version: "encryptionkey@0"
I20260929 11:20:59.309662 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:20:59.312825 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:59.314052 4720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:20:59.314204 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "0b977a7942034f8e9b3d0febe490414f"
format_stamp: "Formatted at 2026-09-29 11:20:59 on dist-test-slave-srst"
server_key: "03d52eb3dae625915c4641b0fb1da50e"
server_key_iv: "f95677dfa76c3964dad66fc82a4e31c4"
server_key_version: "encryptionkey@0"
I20260929 11:20:59.314430 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:59.345870 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:59.350080 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:59.350350 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:59.351297 4720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:59.352741 4720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:59.352849 4720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:59.352980 4720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:59.353075 4720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:59.387766 4720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45053
I20260929 11:20:59.387856 4846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45053 every 8 connection(s)
I20260929 11:20:59.388710 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:20:59.389875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4720
I20260929 11:20:59.390127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:20:59.390648 388 external_mini_cluster.cc:1428] Setting key 29ff0499f0cc0fbb766c6b9ad1378f24
Sep 29 11:20:59 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:20:59.411871 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.391745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51847 (local address 127.0.97.62:44373)
0929 11:20:59.391928 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:59.391933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:59.393293 (+ 1360us) server_negotiation.cc:407] Connection header received
0929 11:20:59.396229 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:59.396232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:59.396292 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:59.396372 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:59.398421 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.399295 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.401309 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.402029 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.402156 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.405499 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:59.405508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:59.405516 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:59.405559 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:59.408685 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:59.409581 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:59.409584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:59.409586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:59.409702 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:59.410203 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:59.410206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:59.410208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:59.410441 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:59.410541 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:59.411472 (+ 931us) server_negotiation.cc:299] Negotiation successful
0929 11:20:59.411630 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:20:59.413146 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.392246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44373 (local address 127.0.97.2:51847)
0929 11:20:59.393115 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:59.393167 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:59.395929 (+ 2762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:59.396651 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:59.396665 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:59.397440 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:59.398211 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.398236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.399447 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.399454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.401179 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.401190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.402231 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.402236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.402442 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.403137 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:59.403160 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:59.405251 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:59.408824 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:59.408833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:59.408851 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:59.409460 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:59.409811 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:59.409818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:59.409821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:59.410088 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:59.410574 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:59.410583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:59.411041 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 11:20:59.411374 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 11:20:59.411827 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":397,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 11:20:59.414579 4847 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44373
I20260929 11:20:59.414897 4847 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:59.415534 4847 heartbeater.cc:511] Master 127.0.97.62:44373 requested a full tablet report, sending...
I20260929 11:20:59.417127 4526 ts_manager.cc:194] Registered new tserver with Master: 0b977a7942034f8e9b3d0febe490414f (127.0.97.2:45053)
I20260929 11:20:59.418067 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:20:59.437882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44373
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:20:59.749720 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:20:59.750088 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:20:59.750269 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:20:59.757555 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:20:59.757714 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:20:59.757898 4852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:20:59.767378 4852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:20:59.768745 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:20:59.770057 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:20:59.782497 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:59.782512 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:20:59.782545 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:20:59.783336 4852 server_base.cc:1034] running on GCE node
I20260929 11:20:59.783938 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:20:59.784914 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:20:59.786129 4852 hybrid_clock.cc:648] HybridClock initialized: now 1790680859786093 us; error 48 us; skew 500 ppm
I20260929 11:20:59.786443 4852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:20:59 dist-test-slave-srst krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:20:59.792733 4852 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:20:59.793865 4852 fs_manager.cc:359] Metadata directory not provided
I20260929 11:20:59.794010 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:20:59.794334 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:20:59.797427 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a274e7939d6403d9445dfb9a61f3c11"
format_stamp: "Formatted at 2026-09-29 11:20:59 on dist-test-slave-srst"
server_key: "bd75a9885f076d8d38366652014bf1f2"
server_key_iv: "42f49721fe1c1cfd833cb743387d71d0"
server_key_version: "encryptionkey@0"
I20260929 11:20:59.798214 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a274e7939d6403d9445dfb9a61f3c11"
format_stamp: "Formatted at 2026-09-29 11:20:59 on dist-test-slave-srst"
server_key: "bd75a9885f076d8d38366652014bf1f2"
server_key_iv: "42f49721fe1c1cfd833cb743387d71d0"
server_key_version: "encryptionkey@0"
I20260929 11:20:59.803234 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 11:20:59.806612 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:59.807878 4852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:20:59.808060 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "7a274e7939d6403d9445dfb9a61f3c11"
format_stamp: "Formatted at 2026-09-29 11:20:59 on dist-test-slave-srst"
server_key: "bd75a9885f076d8d38366652014bf1f2"
server_key_iv: "42f49721fe1c1cfd833cb743387d71d0"
server_key_version: "encryptionkey@0"
I20260929 11:20:59.808274 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:20:59.842092 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:20:59.846117 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:20:59.846392 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:20:59.847330 4852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:20:59.848644 4852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:20:59.848758 4852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:59.848891 4852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:20:59.848984 4852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:20:59.881659 4852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33591
I20260929 11:20:59.881704 4978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33591 every 8 connection(s)
I20260929 11:20:59.882598 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:20:59.885778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 4852
I20260929 11:20:59.885993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:20:59.886539 388 external_mini_cluster.cc:1428] Setting key 975f83a2752d47a7121c4c782b61dbd8
Sep 29 11:20:59 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:20:59.902715 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.885227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60209 (local address 127.0.97.62:44373)
0929 11:20:59.885503 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:59.885512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:59.886851 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 11:20:59.889308 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:59.889313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:59.889398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:59.889510 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:59.891032 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.891741 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.893275 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.893655 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.893747 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.897024 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:59.897034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:59.897037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:59.897066 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:59.899575 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:59.900358 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:59.900364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:59.900367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:59.900486 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:59.900959 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:59.900963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:59.900965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:59.901322 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:59.901447 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:59.902291 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 11:20:59.902453 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:20:59.903645 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.885752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44373 (local address 127.0.97.3:60209)
0929 11:20:59.886662 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:59.886715 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:59.889057 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:59.889780 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:59.889797 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:59.890332 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:20:59.890894 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.890910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.891862 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.891867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.893161 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.893171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.893833 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.893837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.894012 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.894617 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:20:59.894637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:20:59.896817 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:20:59.899684 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:59.899690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:59.899702 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:59.900235 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:59.900634 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:20:59.900640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:20:59.900648 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:20:59.900825 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:20:59.901452 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:20:59.901461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:20:59.901878 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 11:20:59.902182 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 11:20:59.902587 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":358,"threads_started":1}
I20260929 11:20:59.905045 4979 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44373
I20260929 11:20:59.905364 4979 heartbeater.cc:464] Registering TS with master...
I20260929 11:20:59.906095 4979 heartbeater.cc:511] Master 127.0.97.62:44373 requested a full tablet report, sending...
I20260929 11:20:59.907604 4526 ts_manager.cc:194] Registered new tserver with Master: 7a274e7939d6403d9445dfb9a61f3c11 (127.0.97.3:33591)
I20260929 11:20:59.908377 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60209
I20260929 11:20:59.914623 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:20:59.931516 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.919497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.0.97.62:44373)
0929 11:20:59.919738 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:59.919748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:59.920132 (+ 384us) server_negotiation.cc:407] Connection header received
0929 11:20:59.920273 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:59.920277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:59.920342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:59.920434 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:20:59.921987 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.922746 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.923956 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.924377 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.924468 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.926661 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:20:59.926666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:20:59.926669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:20:59.926698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:20:59.929300 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:59.929849 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:59.929852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:59.929855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:59.929965 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:20:59.930270 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:20:59.930273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:20:59.930275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:20:59.930556 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:20:59.930726 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:59.931017 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:20:59.931166 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:20:59.935676 4714 heartbeater.cc:503] Master 127.0.97.62:44373 was elected leader, sending a full tablet report...
I20260929 11:20:59.935369 4526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:20:59.942463 4526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:20:59.978797 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.955115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:55994)
0929 11:20:59.966905 (+ 11790us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:59.966954 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:59.967156 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:59.969868 (+ 2712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:59.969880 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:59.969922 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:59.970510 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.970527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.973346 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.973353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.975891 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.975901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.977715 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.977721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.977941 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.977986 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 11:20:59.978080 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 11:20:59.978262 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11469,"mutex_wait_us":85,"thread_start_us":14053,"threads_started":1}
I20260929 11:20:59.979285 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.955610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33591 (local address 127.0.0.1:48430)
0929 11:20:59.969547 (+ 13937us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:59.969573 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:59.969743 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:59.971035 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:59.971044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:59.971081 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:59.971635 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.971651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.974046 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.974053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.976498 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.976514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.978334 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.978340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.978564 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.978596 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:20:59.978700 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 11:20:59.978852 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13740,"mutex_wait_us":253,"thread_start_us":11759,"threads_started":1}
I20260929 11:20:59.979321 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.955346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.0.97.1:35443)
0929 11:20:59.969134 (+ 13788us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:59.969145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:59.969177 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:20:59.969271 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:59.969278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:59.969583 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:59.970730 (+ 1147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:59.971016 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.973151 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.976034 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.977491 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.978567 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.978774 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:59.978841 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 11:20:59.979018 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13582,"thread_start_us":13440,"threads_started":1}
I20260929 11:20:59.980172 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.955928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.97.3:33591)
0929 11:20:59.968583 (+ 12655us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:59.968613 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:59.969914 (+ 1301us) server_negotiation.cc:407] Connection header received
0929 11:20:59.970049 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:59.970076 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:59.970375 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:59.970627 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:59.971937 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.973872 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.976691 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.978107 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.978230 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.978448 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:59.979705 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0929 11:20:59.979857 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12435,"thread_start_us":12258,"threads_started":1}
I20260929 11:20:59.982904 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.958411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45053 (local address 127.0.0.1:48166)
0929 11:20:59.973700 (+ 15289us) negotiation.cc:107] Waiting for socket to connect
0929 11:20:59.973718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:20:59.973829 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:20:59.975049 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:20:59.975053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:20:59.975082 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:20:59.975473 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.975483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.977870 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.977875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.979937 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:20:59.979950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.981990 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.981996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:20:59.982202 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.982237 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:20:59.982359 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 11:20:59.982480 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15155,"mutex_wait_us":1,"thread_start_us":14962,"threads_started":1}
I20260929 11:20:59.982914 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:20:59.958661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.0.97.2:45053)
0929 11:20:59.974485 (+ 15824us) server_negotiation.cc:206] Beginning negotiation
0929 11:20:59.974494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:20:59.974522 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:20:59.974626 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:20:59.974634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:20:59.974849 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:20:59.975050 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:20:59.975642 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.977734 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.980100 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:20:59.981499 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:20:59.981639 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:20:59.981827 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 11:20:59.982434 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 11:20:59.982651 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15660,"thread_start_us":15518,"threads_started":1}
I20260929 11:20:59.983567 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 33abdf52c4634852987b1ec29f936a2b (DEFAULT_TABLE table=test-table [id=afd40bd3013e474ea42c5597e41ec297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:59.983724 4914 tablet_service.cc:1467] Processing CreateTablet for tablet 33abdf52c4634852987b1ec29f936a2b (DEFAULT_TABLE table=test-table [id=afd40bd3013e474ea42c5597e41ec297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:59.984977 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33abdf52c4634852987b1ec29f936a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:59.985180 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33abdf52c4634852987b1ec29f936a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:59.986315 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 33abdf52c4634852987b1ec29f936a2b (DEFAULT_TABLE table=test-table [id=afd40bd3013e474ea42c5597e41ec297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:20:59.987620 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33abdf52c4634852987b1ec29f936a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:20:59.995900 4997 tablet_bootstrap.cc:492] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: Bootstrap starting.
I20260929 11:21:00.004304 4997 tablet_bootstrap.cc:654] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:00.005362 4998 tablet_bootstrap.cc:492] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Bootstrap starting.
I20260929 11:21:00.005748 4997 log.cc:824] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:00.009821 4998 tablet_bootstrap.cc:654] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:00.011338 4998 log.cc:824] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:00.014374 4997 tablet_bootstrap.cc:492] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: No bootstrap required, opened a new log
I20260929 11:21:00.012636 4999 tablet_bootstrap.cc:492] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Bootstrap starting.
I20260929 11:21:00.014647 4997 ts_tablet_manager.cc:1397] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20260929 11:21:00.014923 4998 tablet_bootstrap.cc:492] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: No bootstrap required, opened a new log
I20260929 11:21:00.015187 4998 ts_tablet_manager.cc:1397] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 11:21:00.018383 4999 tablet_bootstrap.cc:654] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:00.019716 4999 log.cc:824] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:00.022886 4999 tablet_bootstrap.cc:492] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: No bootstrap required, opened a new log
I20260929 11:21:00.023146 4999 ts_tablet_manager.cc:1397] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260929 11:21:00.023147 4997 raft_consensus.cc:348] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.023532 4997 raft_consensus.cc:374] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:00.023633 4997 raft_consensus.cc:729] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc716d7ec3474ef09d07cea3585a7674, State: Initialized, Role: FOLLOWER
I20260929 11:21:00.024219 4997 consensus_queue.cc:260] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.025743 4997 ts_tablet_manager.cc:1428] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260929 11:21:00.025841 4998 raft_consensus.cc:348] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.026268 4998 raft_consensus.cc:374] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:00.026376 4998 raft_consensus.cc:729] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a274e7939d6403d9445dfb9a61f3c11, State: Initialized, Role: FOLLOWER
I20260929 11:21:00.026995 4998 consensus_queue.cc:260] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.028010 4979 heartbeater.cc:503] Master 127.0.97.62:44373 was elected leader, sending a full tablet report...
I20260929 11:21:00.028703 4998 ts_tablet_manager.cc:1428] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 11:21:00.033610 4999 raft_consensus.cc:348] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.033947 4999 raft_consensus.cc:374] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:00.034037 4999 raft_consensus.cc:729] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b977a7942034f8e9b3d0febe490414f, State: Initialized, Role: FOLLOWER
I20260929 11:21:00.034627 4999 consensus_queue.cc:260] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.035589 4847 heartbeater.cc:503] Master 127.0.97.62:44373 was elected leader, sending a full tablet report...
I20260929 11:21:00.036160 4999 ts_tablet_manager.cc:1428] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260929 11:21:00.134099 4980 tablet.cc:2369] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:21:00.140702 4848 tablet.cc:2369] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:21:00.157469 4715 tablet.cc:2369] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:00.209311 5005 raft_consensus.cc:482] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:00.209662 5005 raft_consensus.cc:504] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.211519 5005 leader_election.cc:288] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc716d7ec3474ef09d07cea3585a7674 (127.0.97.1:35443), 7a274e7939d6403d9445dfb9a61f3c11 (127.0.97.3:33591)
I20260929 11:21:00.218921 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.211997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49121 (local address 127.0.97.1:35443)
0929 11:21:00.212285 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:00.212294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:00.213118 (+ 824us) server_negotiation.cc:407] Connection header received
0929 11:21:00.213340 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:00.213344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:00.213402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:00.213505 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:00.214044 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.214765 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.216884 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.217880 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.218060 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.218193 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:00.218535 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 11:21:00.218655 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 11:21:00.219085 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.212018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.97.2:49121)
0929 11:21:00.212980 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.213004 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:00.213175 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:00.213522 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:00.213526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:00.213550 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:00.213894 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:00.213903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.214901 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.214915 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:00.216719 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:00.216747 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.218198 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.218206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:00.218367 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.218390 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:21:00.218505 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:21:00.218644 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":103,"thread_start_us":595,"threads_started":1}
I20260929 11:21:00.219205 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.211985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50919 (local address 127.0.97.3:33591)
0929 11:21:00.212261 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:00.212268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:00.213248 (+ 980us) server_negotiation.cc:407] Connection header received
0929 11:21:00.213441 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:00.213446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:00.213513 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:00.213614 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:00.214217 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.214910 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.216841 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.217998 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.218157 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.218268 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:00.218802 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 11:21:00.218944 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:21:00.219319 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.212113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33591 (local address 127.0.97.2:50919)
0929 11:21:00.213107 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.213125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:00.213315 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:00.213629 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:00.213633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:00.213671 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:00.214078 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:00.214086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.215047 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.215052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:00.216688 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:00.216700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.218216 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.218222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:00.218626 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.218650 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:21:00.218764 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 11:21:00.218898 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":506,"threads_started":1}
I20260929 11:21:00.220105 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33abdf52c4634852987b1ec29f936a2b" candidate_uuid: "0b977a7942034f8e9b3d0febe490414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc716d7ec3474ef09d07cea3585a7674" is_pre_election: true
I20260929 11:21:00.220263 4934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33abdf52c4634852987b1ec29f936a2b" candidate_uuid: "0b977a7942034f8e9b3d0febe490414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a274e7939d6403d9445dfb9a61f3c11" is_pre_election: true
I20260929 11:21:00.220685 4669 raft_consensus.cc:2454] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b977a7942034f8e9b3d0febe490414f in term 0.
I20260929 11:21:00.220772 4934 raft_consensus.cc:2454] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b977a7942034f8e9b3d0febe490414f in term 0.
I20260929 11:21:00.221360 4737 leader_election.cc:302] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b977a7942034f8e9b3d0febe490414f, dc716d7ec3474ef09d07cea3585a7674; no voters:
I20260929 11:21:00.221763 5005 raft_consensus.cc:2781] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:00.221903 5005 raft_consensus.cc:482] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:00.222008 5005 raft_consensus.cc:3037] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:00.224123 5005 raft_consensus.cc:504] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.224822 5005 leader_election.cc:288] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [CANDIDATE]: Term 1 election: Requested vote from peers dc716d7ec3474ef09d07cea3585a7674 (127.0.97.1:35443), 7a274e7939d6403d9445dfb9a61f3c11 (127.0.97.3:33591)
I20260929 11:21:00.225260 4934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33abdf52c4634852987b1ec29f936a2b" candidate_uuid: "0b977a7942034f8e9b3d0febe490414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a274e7939d6403d9445dfb9a61f3c11"
I20260929 11:21:00.225260 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33abdf52c4634852987b1ec29f936a2b" candidate_uuid: "0b977a7942034f8e9b3d0febe490414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc716d7ec3474ef09d07cea3585a7674"
I20260929 11:21:00.225466 4934 raft_consensus.cc:3037] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:00.225466 4669 raft_consensus.cc:3037] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:00.228562 4669 raft_consensus.cc:2454] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b977a7942034f8e9b3d0febe490414f in term 1.
I20260929 11:21:00.228562 4934 raft_consensus.cc:2454] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b977a7942034f8e9b3d0febe490414f in term 1.
I20260929 11:21:00.229004 4735 leader_election.cc:302] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b977a7942034f8e9b3d0febe490414f, 7a274e7939d6403d9445dfb9a61f3c11; no voters:
I20260929 11:21:00.229300 5005 raft_consensus.cc:2781] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:00.229728 5005 raft_consensus.cc:686] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 1 LEADER]: Becoming Leader. State: Replica: 0b977a7942034f8e9b3d0febe490414f, State: Running, Role: LEADER
I20260929 11:21:00.230218 5005 consensus_queue.cc:237] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } }
I20260929 11:21:00.234972 4526 catalog_manager.cc:5495] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b977a7942034f8e9b3d0febe490414f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0b977a7942034f8e9b3d0febe490414f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b977a7942034f8e9b3d0febe490414f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45053 } health_report { overall_health: HEALTHY } } }
I20260929 11:21:00.274649 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.269311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.0.97.2:45053)
0929 11:21:00.269517 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:00.269524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:00.269616 (+ 92us) server_negotiation.cc:407] Connection header received
0929 11:21:00.269860 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:00.269864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:00.269939 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:00.270113 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:00.270471 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.271163 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.272467 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.272898 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.273041 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.273531 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:00.274022 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:00.274188 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:00.274237 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 11:21:00.274323 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:21:00.282984 4934 raft_consensus.cc:1261] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 1 FOLLOWER]: Refusing update from remote peer 0b977a7942034f8e9b3d0febe490414f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:00.282986 4669 raft_consensus.cc:1261] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 1 FOLLOWER]: Refusing update from remote peer 0b977a7942034f8e9b3d0febe490414f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:00.284104 5008 consensus_queue.cc:1035] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a274e7939d6403d9445dfb9a61f3c11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:00.284479 5005 consensus_queue.cc:1035] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc716d7ec3474ef09d07cea3585a7674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:00.296059 5011 mvcc.cc:205] Tried to move back new op lower bound from 7334628803707666432 to 7334628803506601984. Current Snapshot: MvccSnapshot[applied={T|T < 7334628803707666432}]
I20260929 11:21:00.309547 4525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:00.309845 4525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:00.313676 4525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08d5895e0a2b423bbe98d3f3f1817ad3: Sending DeleteTablet for 3 replicas of tablet 33abdf52c4634852987b1ec29f936a2b
I20260929 11:21:00.314786 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 33abdf52c4634852987b1ec29f936a2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55994
I20260929 11:21:00.314785 4914 tablet_service.cc:1514] Processing DeleteTablet for tablet 33abdf52c4634852987b1ec29f936a2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48430
I20260929 11:21:00.315033 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 33abdf52c4634852987b1ec29f936a2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48166
I20260929 11:21:00.317075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4587
I20260929 11:21:00.327766 5023 tablet_replica.cc:330] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: stopping tablet replica
I20260929 11:21:00.328239 5023 raft_consensus.cc:2229] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:00.328511 5021 tablet_replica.cc:330] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: stopping tablet replica
I20260929 11:21:00.328922 5023 raft_consensus.cc:2258] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:00.328984 5021 raft_consensus.cc:2229] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:00.329470 5021 raft_consensus.cc:2258] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:00.331701 5021 ts_tablet_manager.cc:1905] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:00.332747 5023 ts_tablet_manager.cc:1905] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:21:00 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:21:00.338960 5021 ts_tablet_manager.cc:1918] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:00.339129 5021 log.cc:1196] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/33abdf52c4634852987b1ec29f936a2b
I20260929 11:21:00.339603 5021 ts_tablet_manager.cc:1939] T 33abdf52c4634852987b1ec29f936a2b P 7a274e7939d6403d9445dfb9a61f3c11: Deleting consensus metadata
I20260929 11:21:00.341671 4510 catalog_manager.cc:4841] TS 7a274e7939d6403d9445dfb9a61f3c11 (127.0.97.3:33591): tablet 33abdf52c4634852987b1ec29f936a2b (table test-table [id=afd40bd3013e474ea42c5597e41ec297]) successfully deleted
I20260929 11:21:00.346036 5022 tablet_replica.cc:330] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674: stopping tablet replica
I20260929 11:21:00.346448 5022 raft_consensus.cc:2229] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:00.346886 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.318238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.0.97.1:35443)
0929 11:21:00.318412 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:00.318419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:00.328209 (+ 9790us) server_negotiation.cc:407] Connection header received
0929 11:21:00.328620 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:00.328626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:00.328720 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:00.328799 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:00.331121 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.332216 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.334153 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.334747 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.334832 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.339923 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:00.339938 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:00.339956 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:00.340004 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:00.343524 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:00.344355 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:00.344365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:00.344369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:00.344506 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:00.344943 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:00.344947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:00.344951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:00.345336 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:00.345613 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:00.346215 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 11:21:00.346413 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:21:00.346951 5022 raft_consensus.cc:2258] T 33abdf52c4634852987b1ec29f936a2b P dc716d7ec3474ef09d07cea3585a7674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:00.348264 4709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:00.352777 5023 ts_tablet_manager.cc:1918] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:00.352988 5023 log.cc:1196] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/33abdf52c4634852987b1ec29f936a2b
I20260929 11:21:00.353534 5023 ts_tablet_manager.cc:1939] T 33abdf52c4634852987b1ec29f936a2b P 0b977a7942034f8e9b3d0febe490414f: Deleting consensus metadata
I20260929 11:21:00.355264 4511 catalog_manager.cc:4841] TS 0b977a7942034f8e9b3d0febe490414f (127.0.97.2:45053): tablet 33abdf52c4634852987b1ec29f936a2b (table test-table [id=afd40bd3013e474ea42c5597e41ec297]) successfully deleted
I20260929 11:21:00.477886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4587
W20260929 11:21:00.500917 4512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35443 (error 108)
I20260929 11:21:00.501130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4720
I20260929 11:21:00.503167 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.502320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57440)
0929 11:21:00.502821 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.502956 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":302,"threads_started":1}
W20260929 11:21:00.503437 4512 catalog_manager.cc:4584] TS dc716d7ec3474ef09d07cea3585a7674 (127.0.97.1:35443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33abdf52c4634852987b1ec29f936a2b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Sep 29 11:21:00 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:21:00.518539 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.501870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.0.97.2:45053)
0929 11:21:00.502068 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:00.502079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:00.502337 (+ 258us) server_negotiation.cc:407] Connection header received
0929 11:21:00.502520 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:00.502525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:00.502634 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:00.502746 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:00.504296 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.505527 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.506715 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.507390 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.507505 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.511512 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:00.511528 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:00.511548 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:00.511590 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:00.515653 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:00.516279 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:00.516294 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:00.516298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:00.516496 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:00.516881 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:00.516887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:00.516892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:00.517337 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:00.517721 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:00.517897 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 11:21:00.518138 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:21:00.519245 4842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:00.557020 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.556561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57448)
0929 11:21:00.556797 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.556887 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 11:21:00.558110 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.557742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57454)
0929 11:21:00.557910 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.557973 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:21:00.618855 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.618408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57468)
0929 11:21:00.618616 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.618709 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 11:21:00.619839 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.619538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57472)
0929 11:21:00.619657 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.619719 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 11:21:00.632062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4720
I20260929 11:21:00.651306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4852
Sep 29 11:21:00 dist-test-slave-srst krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:21:00.664822 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.652034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.97.3:33591)
0929 11:21:00.652213 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:00.652217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:00.652235 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:21:00.652397 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:00.652401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:00.652459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:00.652538 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:00.653801 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.654507 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.655634 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:00.656012 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:00.656106 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:00.660055 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:00.660075 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:00.660093 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:00.660123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:00.662509 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:00.663111 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:00.663118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:00.663120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:00.663247 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:00.663541 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:00.663544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:00.663546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:00.663851 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:00.664060 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:00.664286 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 11:21:00.664446 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:21:00.665434 4974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:00.700268 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.699721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57480)
0929 11:21:00.699956 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.700089 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 11:21:00.701354 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.700918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57488)
0929 11:21:00.701118 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.701199 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 11:21:00.752575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4852
I20260929 11:21:00.768537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 4495
I20260929 11:21:00.769702 4556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:00.876087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 4495
I20260929 11:21:00.876120 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:00.875614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35443 (local address 127.0.0.1:57496)
0929 11:21:00.875782 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:00.875915 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
2026-09-29T11:21:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4214 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T11:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:00Z Disabled control of system clock
I20260929 11:21:00.942070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41675
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:21:01.252725 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:01.253011 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:01.253118 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:01.260089 5034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:01.260195 5034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:01.260242 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:01.260298 5034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:01.260356 5034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:01.269309 5034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:01.270596 5034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:01.272289 5034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:01.284128 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:01.284205 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:01.284371 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:01.284265 5034 server_base.cc:1034] running on GCE node
I20260929 11:21:01.285470 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:01.286731 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:01.287935 5034 hybrid_clock.cc:648] HybridClock initialized: now 1790680861287867 us; error 61 us; skew 500 ppm
I20260929 11:21:01.288321 5034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:01.290549 5034 webserver.cc:460] Webserver started at http://127.0.97.62:33355/ using document root <none> and password file <none>
I20260929 11:21:01.291239 5034 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:01.291352 5034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:01.291702 5034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:01.294574 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "197bec4c70dc45e18b030bd504c930b5"
format_stamp: "Formatted at 2026-09-29 11:21:01 on dist-test-slave-srst"
server_key: "435e497913c97fb955b660dd96377021"
server_key_iv: "c71a443b69a7ded40d93e9553fcd677e"
server_key_version: "encryptionkey@0"
I20260929 11:21:01.295331 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "197bec4c70dc45e18b030bd504c930b5"
format_stamp: "Formatted at 2026-09-29 11:21:01 on dist-test-slave-srst"
server_key: "435e497913c97fb955b660dd96377021"
server_key_iv: "c71a443b69a7ded40d93e9553fcd677e"
server_key_version: "encryptionkey@0"
I20260929 11:21:01.300251 5034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:21:01.303563 5048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:01.304934 5034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:01.305112 5034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "197bec4c70dc45e18b030bd504c930b5"
format_stamp: "Formatted at 2026-09-29 11:21:01 on dist-test-slave-srst"
server_key: "435e497913c97fb955b660dd96377021"
server_key_iv: "c71a443b69a7ded40d93e9553fcd677e"
server_key_version: "encryptionkey@0"
I20260929 11:21:01.305302 5034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:01.310297 5034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:01.310997 5034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:01.311290 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:01.328678 5034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41675
I20260929 11:21:01.328683 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41675 every 8 connection(s)
I20260929 11:21:01.330045 5034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:01.334121 5100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:01.339370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5034
I20260929 11:21:01.339574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:01.340167 388 external_mini_cluster.cc:1428] Setting key 6974635339e355937f9c4af7bc1d5a0b
I20260929 11:21:01.344499 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5: Bootstrap starting.
I20260929 11:21:01.348008 5100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:01.348351 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:01.342246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.0.97.62:41675)
0929 11:21:01.343235 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:01.343245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:01.343286 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:21:01.344910 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:01.344932 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:01.345340 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:01.345650 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:01.346764 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:01.346770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:01.346782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:01.346796 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:01.346855 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:01.346870 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:01.347024 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:01.347171 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 11:21:01.347421 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":356,"threads_started":1}
I20260929 11:21:01.349334 5100 log.cc:824] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:01.352131 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5: No bootstrap required, opened a new log
I20260929 11:21:01.359062 5100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197bec4c70dc45e18b030bd504c930b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260929 11:21:01.359385 5100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:01.359473 5100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197bec4c70dc45e18b030bd504c930b5, State: Initialized, Role: FOLLOWER
I20260929 11:21:01.359934 5100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197bec4c70dc45e18b030bd504c930b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260929 11:21:01.360131 5100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:01.360220 5100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:01.360329 5100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:01.362345 5100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197bec4c70dc45e18b030bd504c930b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260929 11:21:01.362758 5100 leader_election.cc:302] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 197bec4c70dc45e18b030bd504c930b5; no voters:
I20260929 11:21:01.363312 5100 leader_election.cc:288] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:01.363466 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:01.364166 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [term 1 LEADER]: Becoming Leader. State: Replica: 197bec4c70dc45e18b030bd504c930b5, State: Running, Role: LEADER
I20260929 11:21:01.364858 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197bec4c70dc45e18b030bd504c930b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260929 11:21:01.365514 5100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:01.367487 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "197bec4c70dc45e18b030bd504c930b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197bec4c70dc45e18b030bd504c930b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } } }
I20260929 11:21:01.367911 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:01.367990 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 197bec4c70dc45e18b030bd504c930b5. Latest consensus state: current_term: 1 leader_uuid: "197bec4c70dc45e18b030bd504c930b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197bec4c70dc45e18b030bd504c930b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } } }
I20260929 11:21:01.368321 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:01.369637 5114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:01.375258 5114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:01.384779 5114 catalog_manager.cc:1347] Generated new cluster ID: 5b48cd9ff5b34d76b29a828492b7e507
I20260929 11:21:01.384917 5114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:01.405588 5114 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:01.406085 5114 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:01.420313 5114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5: Generated new TSK 0
I20260929 11:21:01.421314 5114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:21:01.431032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:21:01.754673 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:01.754931 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:01.755048 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:01.762053 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:01.762149 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:01.762370 5124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:01.771770 5124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:01.773088 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:01.774812 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:01.788229 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:01.788197 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:01.788425 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:01.789110 5124 server_base.cc:1034] running on GCE node
I20260929 11:21:01.789887 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:01.790906 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:01.792155 5124 hybrid_clock.cc:648] HybridClock initialized: now 1790680861792100 us; error 68 us; skew 500 ppm
I20260929 11:21:01.792514 5124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:01.795073 5124 webserver.cc:460] Webserver started at http://127.0.97.1:34061/ using document root <none> and password file <none>
I20260929 11:21:01.795814 5124 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:01.795957 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:01.796301 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:01.799518 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "9745846ffc1744b8904fdb428d2aeebb"
format_stamp: "Formatted at 2026-09-29 11:21:01 on dist-test-slave-srst"
server_key: "15cb191881911294eb45b3462bb95494"
server_key_iv: "5dd52f3989707cffa3a4366b09b80227"
server_key_version: "encryptionkey@0"
I20260929 11:21:01.800417 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9745846ffc1744b8904fdb428d2aeebb"
format_stamp: "Formatted at 2026-09-29 11:21:01 on dist-test-slave-srst"
server_key: "15cb191881911294eb45b3462bb95494"
server_key_iv: "5dd52f3989707cffa3a4366b09b80227"
server_key_version: "encryptionkey@0"
I20260929 11:21:01.805887 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:21:01.809603 5138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:01.811034 5124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:01.811233 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "9745846ffc1744b8904fdb428d2aeebb"
format_stamp: "Formatted at 2026-09-29 11:21:01 on dist-test-slave-srst"
server_key: "15cb191881911294eb45b3462bb95494"
server_key_iv: "5dd52f3989707cffa3a4366b09b80227"
server_key_version: "encryptionkey@0"
I20260929 11:21:01.811491 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:01.817152 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:01.817912 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:01.818176 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:01.819034 5124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:01.820384 5124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:01.820501 5124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:01.820698 5124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:01.820788 5124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:01.853735 5124 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41605
I20260929 11:21:01.853796 5250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41605 every 8 connection(s)
I20260929 11:21:01.855108 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:01.861325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5124
I20260929 11:21:01.861534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:01.862156 388 external_mini_cluster.cc:1428] Setting key 3fe13332abbb38bec16f996c01937ebe
I20260929 11:21:01.863559 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:01.857724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42283 (local address 127.0.97.62:41675)
0929 11:21:01.858125 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:01.858129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:01.859384 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 11:21:01.861149 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:01.861154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:01.861229 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:01.861355 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:01.862804 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:01.862809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:01.862814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:01.862831 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:01.862880 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:01.862893 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:01.863021 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:01.863209 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 11:21:01.863271 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":209,"threads_started":1}
I20260929 11:21:01.864512 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:01.858314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41675 (local address 127.0.97.1:42283)
0929 11:21:01.859230 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:01.859268 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:01.860873 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:01.861600 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:01.861632 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:01.861747 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:01.862458 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:01.862460 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:01.862503 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:01.862636 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:01.862987 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:01.862997 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:01.863007 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:21:01.863293 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 11:21:01.863548 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":280,"threads_started":1}
I20260929 11:21:01.864872 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 11:21:01.866179 5251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41675
I20260929 11:21:01.866489 5251 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:01.867136 5251 heartbeater.cc:511] Master 127.0.97.62:41675 requested a full tablet report, sending...
I20260929 11:21:01.869652 5065 ts_manager.cc:194] Registered new tserver with Master: 9745846ffc1744b8904fdb428d2aeebb (127.0.97.1:41605)
W20260929 11:21:02.181396 5255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:02.181674 5255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:02.181763 5255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:02.188820 5255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:02.188923 5255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:02.189033 5255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:02.197947 5255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:02.199178 5255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:02.201082 5255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:02.213706 5263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:02.213798 5261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:02.214071 5260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:02.214490 5255 server_base.cc:1034] running on GCE node
I20260929 11:21:02.215102 5255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:02.215987 5255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:02.217175 5255 hybrid_clock.cc:648] HybridClock initialized: now 1790680862217132 us; error 48 us; skew 500 ppm
I20260929 11:21:02.217469 5255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:02.219635 5255 webserver.cc:460] Webserver started at http://127.0.97.2:45299/ using document root <none> and password file <none>
I20260929 11:21:02.220302 5255 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:02.220415 5255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:02.220796 5255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:02.223517 5255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "a3af0348284b48ef9d194aa897936e77"
format_stamp: "Formatted at 2026-09-29 11:21:02 on dist-test-slave-srst"
server_key: "677a07c922b00595285dce1bd1abeebe"
server_key_iv: "b7d1fd74419ce3c6e5c69f370894127e"
server_key_version: "encryptionkey@0"
I20260929 11:21:02.224285 5255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a3af0348284b48ef9d194aa897936e77"
format_stamp: "Formatted at 2026-09-29 11:21:02 on dist-test-slave-srst"
server_key: "677a07c922b00595285dce1bd1abeebe"
server_key_iv: "b7d1fd74419ce3c6e5c69f370894127e"
server_key_version: "encryptionkey@0"
I20260929 11:21:02.229152 5255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:21:02.232564 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:02.233994 5255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:02.234179 5255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "a3af0348284b48ef9d194aa897936e77"
format_stamp: "Formatted at 2026-09-29 11:21:02 on dist-test-slave-srst"
server_key: "677a07c922b00595285dce1bd1abeebe"
server_key_iv: "b7d1fd74419ce3c6e5c69f370894127e"
server_key_version: "encryptionkey@0"
I20260929 11:21:02.234416 5255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:02.240088 5255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:02.240906 5255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:02.241176 5255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:02.242033 5255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:02.243351 5255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:02.243470 5255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:02.243608 5255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:02.243696 5255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:02.275919 5255 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36875
I20260929 11:21:02.275976 5381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36875 every 8 connection(s)
I20260929 11:21:02.277240 5255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:02.284332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5255
I20260929 11:21:02.284513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:02.284983 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.279610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40123 (local address 127.0.97.62:41675)
0929 11:21:02.279816 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.279823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.281273 (+ 1450us) server_negotiation.cc:407] Connection header received
0929 11:21:02.282837 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.282840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.282887 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.282975 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.284243 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.284246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.284249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.284264 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.284291 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.284302 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.284402 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.284715 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.284773 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:21:02.285090 388 external_mini_cluster.cc:1428] Setting key 4d502de3089a2fbf0277e431fb81c494
I20260929 11:21:02.286170 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.280198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41675 (local address 127.0.97.2:40123)
0929 11:21:02.281117 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.281158 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.282599 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.283167 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.283174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.283277 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.283885 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.283888 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.283952 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.284071 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.284401 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.284413 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.284423 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.284774 (+ 351us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.285022 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":259,"threads_started":1}
I20260929 11:21:02.287496 5382 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41675
I20260929 11:21:02.287811 5382 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:02.288071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 11:21:02.288466 5382 heartbeater.cc:511] Master 127.0.97.62:41675 requested a full tablet report, sending...
I20260929 11:21:02.289927 5065 ts_manager.cc:194] Registered new tserver with Master: a3af0348284b48ef9d194aa897936e77 (127.0.97.2:36875)
W20260929 11:21:02.616223 5385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:02.616508 5385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:02.616688 5385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:02.623974 5385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:02.624081 5385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:02.624197 5385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:02.633944 5385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:02.635192 5385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:02.637028 5385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:02.650985 5390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:02.650938 5393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:02.650933 5391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:02.651867 5385 server_base.cc:1034] running on GCE node
I20260929 11:21:02.652415 5385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:02.653345 5385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:02.654534 5385 hybrid_clock.cc:648] HybridClock initialized: now 1790680862654486 us; error 50 us; skew 500 ppm
I20260929 11:21:02.654837 5385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:02.659849 5385 webserver.cc:460] Webserver started at http://127.0.97.3:44861/ using document root <none> and password file <none>
I20260929 11:21:02.660522 5385 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:02.660652 5385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:02.660996 5385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:02.663789 5385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "739085aac3304b639147f0d418311caf"
format_stamp: "Formatted at 2026-09-29 11:21:02 on dist-test-slave-srst"
server_key: "55421324829f698053dc3ca41df9f15a"
server_key_iv: "d6e8e852580f292bcff51f632cc8d68e"
server_key_version: "encryptionkey@0"
I20260929 11:21:02.664568 5385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "739085aac3304b639147f0d418311caf"
format_stamp: "Formatted at 2026-09-29 11:21:02 on dist-test-slave-srst"
server_key: "55421324829f698053dc3ca41df9f15a"
server_key_iv: "d6e8e852580f292bcff51f632cc8d68e"
server_key_version: "encryptionkey@0"
I20260929 11:21:02.669669 5385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:21:02.673115 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:02.674409 5385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:21:02.674572 5385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "739085aac3304b639147f0d418311caf"
format_stamp: "Formatted at 2026-09-29 11:21:02 on dist-test-slave-srst"
server_key: "55421324829f698053dc3ca41df9f15a"
server_key_iv: "d6e8e852580f292bcff51f632cc8d68e"
server_key_version: "encryptionkey@0"
I20260929 11:21:02.674799 5385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:02.680250 5385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:02.681126 5385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:02.681376 5385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:02.682245 5385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:02.683516 5385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:02.683604 5385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:02.683735 5385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:02.683828 5385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:02.717095 5385 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43565
I20260929 11:21:02.717154 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43565 every 8 connection(s)
I20260929 11:21:02.718384 5385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:02.726456 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.720989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53433 (local address 127.0.97.62:41675)
0929 11:21:02.721154 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.721159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.722539 (+ 1380us) server_negotiation.cc:407] Connection header received
0929 11:21:02.724188 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.724194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.724261 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.724380 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.725745 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.725750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.725755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.725781 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.725817 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.725828 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.725944 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.726131 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.726196 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:21:02.727373 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.721516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41675 (local address 127.0.97.3:53433)
0929 11:21:02.722386 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.722420 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.723924 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.724667 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.724680 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.724804 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.725454 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.725456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.725493 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.725571 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.725929 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.725939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.725946 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.726210 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.726462 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":257,"threads_started":1}
I20260929 11:21:02.727385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5385
I20260929 11:21:02.727536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:02.728051 388 external_mini_cluster.cc:1428] Setting key 7f68390ea8b543aa79f6168e37d3db70
I20260929 11:21:02.728834 5512 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41675
I20260929 11:21:02.729123 5512 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:02.729785 5512 heartbeater.cc:511] Master 127.0.97.62:41675 requested a full tablet report, sending...
I20260929 11:21:02.731133 5065 ts_manager.cc:194] Registered new tserver with Master: 739085aac3304b639147f0d418311caf (127.0.97.3:43565)
I20260929 11:21:02.743072 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:21:02.749990 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.747641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.0.97.62:41675)
0929 11:21:02.747806 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.747811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.748310 (+ 499us) server_negotiation.cc:407] Connection header received
0929 11:21:02.748445 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.748449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.748502 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.748652 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.749500 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.749505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.749508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.749523 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.749555 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.749568 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.749679 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.749730 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.749775 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:21:02.753952 5065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:02.764237 5065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:02.786834 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.778662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50230)
0929 11:21:02.779861 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.779895 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.780034 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.781739 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.781751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.781886 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.785269 (+ 3383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.785275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.785312 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.785407 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.786016 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.786027 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.786035 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.786223 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.786325 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":833,"threads_started":1}
I20260929 11:21:02.787925 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.778863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.97.1:41605)
0929 11:21:02.779331 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.779340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.780024 (+ 684us) server_negotiation.cc:407] Connection header received
0929 11:21:02.780173 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.780180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.780456 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.780656 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.785639 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.785652 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.785670 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.785690 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.785781 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.785808 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.785957 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.787366 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.787489 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":267,"threads_started":1}
I20260929 11:21:02.791860 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.781220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36875 (local address 127.0.0.1:54884)
0929 11:21:02.787065 (+ 5845us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.787085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.787178 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.789947 (+ 2769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.789954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.789991 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.790578 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.790581 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.790598 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.790640 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.791355 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.791361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.791369 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.791484 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.791539 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5735,"threads_started":1}
I20260929 11:21:02.792461 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.781324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.0.97.2:36875)
0929 11:21:02.789326 (+ 8002us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.789338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.789377 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:21:02.789460 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.789472 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.789758 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.789957 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.791082 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.791094 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.791112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.791133 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.791198 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.791221 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.791389 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.792076 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.792169 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7813,"thread_start_us":7661,"threads_started":1}
I20260929 11:21:02.797389 5317 tablet_service.cc:1467] Processing CreateTablet for tablet 5f62ca861ebf49aeb6920ece41499402 (DEFAULT_TABLE table=test-table [id=06c65c05f3244492b829d0ea9bb8f1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:02.797678 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 5f62ca861ebf49aeb6920ece41499402 (DEFAULT_TABLE table=test-table [id=06c65c05f3244492b829d0ea9bb8f1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:02.799038 5317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f62ca861ebf49aeb6920ece41499402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:02.799260 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f62ca861ebf49aeb6920ece41499402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:02.816509 5530 tablet_bootstrap.cc:492] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Bootstrap starting.
I20260929 11:21:02.819533 5530 tablet_bootstrap.cc:654] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:02.820405 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.780347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43565 (local address 127.0.0.1:53596)
0929 11:21:02.780897 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.780915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.781003 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.813955 (+ 32952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.813960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.814004 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.814843 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.814846 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.814866 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.814949 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.819689 (+ 4740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.819695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.819700 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.819871 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.819943 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
I20260929 11:21:02.820814 5531 tablet_bootstrap.cc:492] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: Bootstrap starting.
I20260929 11:21:02.820863 5530 log.cc:824] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:02.822674 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.780643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.97.3:43565)
0929 11:21:02.813251 (+ 32608us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.813268 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.813305 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:21:02.813403 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.813411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.813679 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.813929 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.816545 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.816552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.816567 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.816652 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.816710 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.816729 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.822031 (+ 5302us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.822156 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.822276 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32441,"thread_start_us":32279,"threads_started":1}
I20260929 11:21:02.823649 5530 tablet_bootstrap.cc:492] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: No bootstrap required, opened a new log
I20260929 11:21:02.823920 5530 ts_tablet_manager.cc:1397] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260929 11:21:02.824303 5531 tablet_bootstrap.cc:654] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:02.825883 5531 log.cc:824] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:02.826565 5447 tablet_service.cc:1467] Processing CreateTablet for tablet 5f62ca861ebf49aeb6920ece41499402 (DEFAULT_TABLE table=test-table [id=06c65c05f3244492b829d0ea9bb8f1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:02.827989 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f62ca861ebf49aeb6920ece41499402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:02.828223 5531 tablet_bootstrap.cc:492] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: No bootstrap required, opened a new log
I20260929 11:21:02.828446 5531 ts_tablet_manager.cc:1397] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 11:21:02.834561 5530 raft_consensus.cc:348] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.835012 5530 raft_consensus.cc:374] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:02.835119 5530 raft_consensus.cc:729] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af0348284b48ef9d194aa897936e77, State: Initialized, Role: FOLLOWER
I20260929 11:21:02.835810 5530 consensus_queue.cc:260] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.835857 5531 raft_consensus.cc:348] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.836198 5531 raft_consensus.cc:374] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:02.836309 5531 raft_consensus.cc:729] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9745846ffc1744b8904fdb428d2aeebb, State: Initialized, Role: FOLLOWER
I20260929 11:21:02.836946 5382 heartbeater.cc:503] Master 127.0.97.62:41675 was elected leader, sending a full tablet report...
I20260929 11:21:02.837738 5530 ts_tablet_manager.cc:1428] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260929 11:21:02.838451 5531 consensus_queue.cc:260] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.839671 5251 heartbeater.cc:503] Master 127.0.97.62:41675 was elected leader, sending a full tablet report...
I20260929 11:21:02.840252 5531 ts_tablet_manager.cc:1428] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20260929 11:21:02.842721 5536 tablet_bootstrap.cc:492] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Bootstrap starting.
I20260929 11:21:02.846118 5536 tablet_bootstrap.cc:654] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:02.847059 5536 log.cc:824] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:02.849639 5536 tablet_bootstrap.cc:492] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: No bootstrap required, opened a new log
I20260929 11:21:02.849903 5536 ts_tablet_manager.cc:1397] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260929 11:21:02.856496 5536 raft_consensus.cc:348] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.856900 5536 raft_consensus.cc:374] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:02.856993 5536 raft_consensus.cc:729] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739085aac3304b639147f0d418311caf, State: Initialized, Role: FOLLOWER
W20260929 11:21:02.857254 5252 tablet.cc:2369] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:02.857673 5536 consensus_queue.cc:260] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.858741 5512 heartbeater.cc:503] Master 127.0.97.62:41675 was elected leader, sending a full tablet report...
I20260929 11:21:02.859213 5536 ts_tablet_manager.cc:1428] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260929 11:21:02.878768 5535 raft_consensus.cc:482] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:02.879055 5535 raft_consensus.cc:504] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.880626 5535 leader_election.cc:288] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 739085aac3304b639147f0d418311caf (127.0.97.3:43565), a3af0348284b48ef9d194aa897936e77 (127.0.97.2:36875)
I20260929 11:21:02.883698 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.880911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48855 (local address 127.0.97.3:43565)
0929 11:21:02.881081 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.881088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.881594 (+ 506us) server_negotiation.cc:407] Connection header received
0929 11:21:02.881750 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.881755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.881821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.881932 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.883020 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.883027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.883032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.883054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.883095 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.883109 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.883266 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.883346 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.883413 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:21:02.883916 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.880870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43565 (local address 127.0.97.1:48855)
0929 11:21:02.881490 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.881511 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.881619 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.882083 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.882093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.882121 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.882772 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.882775 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.882790 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.882848 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.883229 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.883234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.883240 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.883383 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.883432 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":306,"threads_started":1}
I20260929 11:21:02.884366 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.881758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51787 (local address 127.0.97.2:36875)
0929 11:21:02.881983 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.881989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.882304 (+ 315us) server_negotiation.cc:407] Connection header received
0929 11:21:02.882724 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.882728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.882771 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.882910 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.883787 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.883803 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.883807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.883827 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.883867 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.883881 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.884004 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.884126 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.884185 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:21:02.885077 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.881612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36875 (local address 127.0.97.1:51787)
0929 11:21:02.882207 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:02.882222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:02.882359 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:02.882934 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:02.882938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:02.882960 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:02.883597 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:02.883600 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:02.883612 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:02.883645 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:02.883981 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:02.883985 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:02.883989 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 11:21:02.884643 (+ 654us) client_negotiation.cc:240] Negotiation successful
0929 11:21:02.884694 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":332,"threads_started":1}
I20260929 11:21:02.884913 5467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f62ca861ebf49aeb6920ece41499402" candidate_uuid: "9745846ffc1744b8904fdb428d2aeebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739085aac3304b639147f0d418311caf" is_pre_election: true
I20260929 11:21:02.885473 5467 raft_consensus.cc:2454] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9745846ffc1744b8904fdb428d2aeebb in term 0.
I20260929 11:21:02.886106 5139 leader_election.cc:302] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739085aac3304b639147f0d418311caf, 9745846ffc1744b8904fdb428d2aeebb; no voters:
I20260929 11:21:02.886094 5337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f62ca861ebf49aeb6920ece41499402" candidate_uuid: "9745846ffc1744b8904fdb428d2aeebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3af0348284b48ef9d194aa897936e77" is_pre_election: true
I20260929 11:21:02.886458 5535 raft_consensus.cc:2781] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:02.886612 5535 raft_consensus.cc:482] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:02.886721 5535 raft_consensus.cc:3037] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:02.886698 5337 raft_consensus.cc:2454] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9745846ffc1744b8904fdb428d2aeebb in term 0.
I20260929 11:21:02.889142 5535 raft_consensus.cc:504] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.889707 5535 leader_election.cc:288] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [CANDIDATE]: Term 1 election: Requested vote from peers 739085aac3304b639147f0d418311caf (127.0.97.3:43565), a3af0348284b48ef9d194aa897936e77 (127.0.97.2:36875)
I20260929 11:21:02.890162 5467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f62ca861ebf49aeb6920ece41499402" candidate_uuid: "9745846ffc1744b8904fdb428d2aeebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739085aac3304b639147f0d418311caf"
I20260929 11:21:02.890280 5337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f62ca861ebf49aeb6920ece41499402" candidate_uuid: "9745846ffc1744b8904fdb428d2aeebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3af0348284b48ef9d194aa897936e77"
I20260929 11:21:02.890349 5467 raft_consensus.cc:3037] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:02.890535 5337 raft_consensus.cc:3037] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:02.892501 5467 raft_consensus.cc:2454] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9745846ffc1744b8904fdb428d2aeebb in term 1.
I20260929 11:21:02.892860 5139 leader_election.cc:302] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739085aac3304b639147f0d418311caf, 9745846ffc1744b8904fdb428d2aeebb; no voters:
I20260929 11:21:02.893006 5337 raft_consensus.cc:2454] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9745846ffc1744b8904fdb428d2aeebb in term 1.
I20260929 11:21:02.893117 5535 raft_consensus.cc:2781] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:02.893595 5535 raft_consensus.cc:686] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [term 1 LEADER]: Becoming Leader. State: Replica: 9745846ffc1744b8904fdb428d2aeebb, State: Running, Role: LEADER
I20260929 11:21:02.894119 5535 consensus_queue.cc:237] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } }
I20260929 11:21:02.898315 5065 catalog_manager.cc:5495] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9745846ffc1744b8904fdb428d2aeebb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9745846ffc1744b8904fdb428d2aeebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9745846ffc1744b8904fdb428d2aeebb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 } health_report { overall_health: UNKNOWN } } }
I20260929 11:21:02.916483 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.914239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.0.97.1:41605)
0929 11:21:02.914406 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.914412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.914579 (+ 167us) server_negotiation.cc:407] Connection header received
0929 11:21:02.914810 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.914814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.914886 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.915018 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.915862 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.915868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.915872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.915888 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.915924 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.915935 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.916047 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.916170 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.916223 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:21:02.923695 5337 raft_consensus.cc:1261] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 1 FOLLOWER]: Refusing update from remote peer 9745846ffc1744b8904fdb428d2aeebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:02.923784 5467 raft_consensus.cc:1261] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 1 FOLLOWER]: Refusing update from remote peer 9745846ffc1744b8904fdb428d2aeebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:02.924755 5541 consensus_queue.cc:1035] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af0348284b48ef9d194aa897936e77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:02.925100 5535 consensus_queue.cc:1035] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "739085aac3304b639147f0d418311caf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:02.949929 5064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:02.950140 5064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:02.953531 5064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 197bec4c70dc45e18b030bd504c930b5: Sending DeleteTablet for 3 replicas of tablet 5f62ca861ebf49aeb6920ece41499402
I20260929 11:21:02.954584 5186 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f62ca861ebf49aeb6920ece41499402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:02 UTC) from {username='slave'} at 127.0.0.1:50230
I20260929 11:21:02.954666 5447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f62ca861ebf49aeb6920ece41499402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:02 UTC) from {username='slave'} at 127.0.0.1:53596
I20260929 11:21:02.954830 5317 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f62ca861ebf49aeb6920ece41499402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:02 UTC) from {username='slave'} at 127.0.0.1:54884
I20260929 11:21:02.957276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5124
I20260929 11:21:02.969029 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:02.958150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.97.1:41605)
0929 11:21:02.958366 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:02.958375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:02.964612 (+ 6237us) server_negotiation.cc:407] Connection header received
0929 11:21:02.965691 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:02.965700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:02.965782 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:02.965887 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:02.967808 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:02.967817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:02.967824 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:02.967850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:02.967896 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:02.967917 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:02.968029 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:02.968487 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 11:21:02.968565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:21:02.969475 5556 tablet_replica.cc:330] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: stopping tablet replica
I20260929 11:21:02.970402 5246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:02.971333 5556 raft_consensus.cc:2229] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:02.971988 5556 raft_consensus.cc:2258] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:02.973953 5555 tablet_replica.cc:330] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: stopping tablet replica
I20260929 11:21:02.974548 5555 raft_consensus.cc:2229] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:02.975059 5556 ts_tablet_manager.cc:1905] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:02.975201 5555 raft_consensus.cc:2258] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:02.977073 5555 ts_tablet_manager.cc:1905] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:02.979990 5556 ts_tablet_manager.cc:1918] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:02.980152 5556 log.cc:1196] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/5f62ca861ebf49aeb6920ece41499402
I20260929 11:21:02.980671 5556 ts_tablet_manager.cc:1939] T 5f62ca861ebf49aeb6920ece41499402 P a3af0348284b48ef9d194aa897936e77: Deleting consensus metadata
I20260929 11:21:02.982134 5049 catalog_manager.cc:4841] TS a3af0348284b48ef9d194aa897936e77 (127.0.97.2:36875): tablet 5f62ca861ebf49aeb6920ece41499402 (table test-table [id=06c65c05f3244492b829d0ea9bb8f1de]) successfully deleted
I20260929 11:21:02.984092 5555 ts_tablet_manager.cc:1918] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:02.984278 5555 log.cc:1196] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/5f62ca861ebf49aeb6920ece41499402
I20260929 11:21:02.984874 5555 ts_tablet_manager.cc:1939] T 5f62ca861ebf49aeb6920ece41499402 P 739085aac3304b639147f0d418311caf: Deleting consensus metadata
I20260929 11:21:02.986689 5049 catalog_manager.cc:4841] TS 739085aac3304b639147f0d418311caf (127.0.97.3:43565): tablet 5f62ca861ebf49aeb6920ece41499402 (table test-table [id=06c65c05f3244492b829d0ea9bb8f1de]) successfully deleted
I20260929 11:21:03.070005 5554 tablet_replica.cc:330] T 5f62ca861ebf49aeb6920ece41499402 P 9745846ffc1744b8904fdb428d2aeebb: stopping tablet replica
I20260929 11:21:03.070175 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5124
W20260929 11:21:03.087365 5049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41605 (error 108)
I20260929 11:21:03.087555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5255
I20260929 11:21:03.089184 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.088800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50256)
0929 11:21:03.088958 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.089065 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:21:03.089461 5049 catalog_manager.cc:4584] TS 9745846ffc1744b8904fdb428d2aeebb (127.0.97.1:41605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f62ca861ebf49aeb6920ece41499402: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
I20260929 11:21:03.090585 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.088429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.0.97.2:36875)
0929 11:21:03.088577 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:03.088585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:03.088688 (+ 103us) server_negotiation.cc:407] Connection header received
0929 11:21:03.088905 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:03.088909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:03.088990 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:03.089095 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:03.089985 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:03.089989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:03.089992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:03.090006 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:03.090034 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:03.090044 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:03.090141 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:03.090206 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 11:21:03.090245 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:21:03.091207 5377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:03.143281 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.142782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50272)
0929 11:21:03.143031 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.143126 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 11:21:03.144302 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.143987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50286)
0929 11:21:03.144140 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.144201 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 11:21:03.179160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5255
I20260929 11:21:03.195389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5385
I20260929 11:21:03.198002 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.196179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.0.97.3:43565)
0929 11:21:03.196356 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:03.196361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:03.196380 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:21:03.196454 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:03.196457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:03.196528 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:03.196649 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:03.197412 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:03.197416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:03.197419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:03.197433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:03.197462 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:03.197473 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:03.197598 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:03.197651 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 11:21:03.197694 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:21:03.198655 5507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:03.205513 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.204958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50290)
0929 11:21:03.205133 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.205253 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 11:21:03.206436 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.206115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50306)
0929 11:21:03.206263 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.206333 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:21:03.285178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5385
I20260929 11:21:03.287201 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.286587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50322)
0929 11:21:03.286844 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.286972 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 11:21:03.288360 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.287862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41605 (local address 127.0.0.1:50330)
0929 11:21:03.288103 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.288204 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 11:21:03.302101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5034
I20260929 11:21:03.303180 5095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:03.408811 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5034
2026-09-29T11:21:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2528 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 11:21:03.446249 388 mini_oidc.cc:152] Starting JWKS server
I20260929 11:21:03.448825 388 webserver.cc:460] Webserver started at https://127.0.0.1:45341/ using document root <none> and password file <none>
I20260929 11:21:03.449151 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:21:03.449790 388 webserver.cc:460] Webserver started at http://127.0.0.1:46833/ using document root <none> and password file <none>
2026-09-29T11:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:03Z Disabled control of system clock
I20260929 11:21:03.475651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42267
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42267
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45341/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/testchainca.cert with env {}
W20260929 11:21:03.791935 5568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:03.792273 5568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:03.792438 5568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:03.799607 5568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:03.799764 5568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:03.799856 5568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:03.799980 5568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:03.800072 5568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:21:03.800163 5568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:21:03.800240 5568 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45341/jwks.json
I20260929 11:21:03.810331 5568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45341/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:03.811682 5568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:03.813024 5568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:03.824728 5576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:03.824858 5573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:03.824805 5574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:03.826058 5568 server_base.cc:1034] running on GCE node
I20260929 11:21:03.826853 5568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:03.828379 5568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:03.829658 5568 hybrid_clock.cc:648] HybridClock initialized: now 1790680863829619 us; error 46 us; skew 500 ppm
I20260929 11:21:03.830057 5568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:03.832487 5568 webserver.cc:460] Webserver started at http://127.0.97.62:40799/ using document root <none> and password file <none>
I20260929 11:21:03.833235 5568 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:03.833395 5568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:03.833738 5568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:03.836830 5568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "a07975b45adf410daeec96d060815247"
format_stamp: "Formatted at 2026-09-29 11:21:03 on dist-test-slave-srst"
server_key: "e97fa7e2c2d9e7d8d8e7606beab2bf15"
server_key_iv: "43ff2e08d68144960014a075603cf39d"
server_key_version: "encryptionkey@0"
I20260929 11:21:03.837653 5568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "a07975b45adf410daeec96d060815247"
format_stamp: "Formatted at 2026-09-29 11:21:03 on dist-test-slave-srst"
server_key: "e97fa7e2c2d9e7d8d8e7606beab2bf15"
server_key_iv: "43ff2e08d68144960014a075603cf39d"
server_key_version: "encryptionkey@0"
I20260929 11:21:03.842760 5568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:21:03.846259 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:03.847599 5568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:21:03.847806 5568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "a07975b45adf410daeec96d060815247"
format_stamp: "Formatted at 2026-09-29 11:21:03 on dist-test-slave-srst"
server_key: "e97fa7e2c2d9e7d8d8e7606beab2bf15"
server_key_iv: "43ff2e08d68144960014a075603cf39d"
server_key_version: "encryptionkey@0"
I20260929 11:21:03.848024 5568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:03.865708 5568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:03.866565 5568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:03.866850 5568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:03.885157 5568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42267
I20260929 11:21:03.885176 5633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42267 every 8 connection(s)
I20260929 11:21:03.886539 5568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:03.890384 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:03.892338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5568
I20260929 11:21:03.892547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:03.893328 388 external_mini_cluster.cc:1428] Setting key c3558dc8e8f3cdf2f2cd4a41c098953f
I20260929 11:21:03.902458 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247: Bootstrap starting.
I20260929 11:21:03.905999 5634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:03.906983 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.895149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.97.62:42267)
0929 11:21:03.896172 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:03.896182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:03.896223 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:21:03.897936 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:03.897962 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:03.898376 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:03.898665 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:03.899226 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:03.900464 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:03.902289 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:03.903059 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:03.903212 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:03.904479 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:03.904487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:03.904504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:03.905129 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:03.905223 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:03.905244 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:03.905344 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:03.905602 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 11:21:03.905974 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":384,"threads_started":1}
I20260929 11:21:03.907157 5634 log.cc:824] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:03.909955 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247: No bootstrap required, opened a new log
I20260929 11:21:03.910595 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:21:03.917037 5634 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07975b45adf410daeec96d060815247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42267 } }
I20260929 11:21:03.917369 5634 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:03.917486 5634 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07975b45adf410daeec96d060815247, State: Initialized, Role: FOLLOWER
I20260929 11:21:03.918062 5634 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07975b45adf410daeec96d060815247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42267 } }
I20260929 11:21:03.918258 5634 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:03.918401 5634 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:03.918555 5634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:03.920828 5634 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07975b45adf410daeec96d060815247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42267 } }
I20260929 11:21:03.921278 5634 leader_election.cc:302] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07975b45adf410daeec96d060815247; no voters:
I20260929 11:21:03.921782 5634 leader_election.cc:288] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:03.921950 5639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:03.922606 5639 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [term 1 LEADER]: Becoming Leader. State: Replica: a07975b45adf410daeec96d060815247, State: Running, Role: LEADER
I20260929 11:21:03.923099 5639 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07975b45adf410daeec96d060815247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42267 } }
I20260929 11:21:03.923743 5634 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:03.925992 5641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07975b45adf410daeec96d060815247. Latest consensus state: current_term: 1 leader_uuid: "a07975b45adf410daeec96d060815247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07975b45adf410daeec96d060815247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42267 } } }
I20260929 11:21:03.926322 5641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:03.926043 5640 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07975b45adf410daeec96d060815247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07975b45adf410daeec96d060815247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42267 } } }
I20260929 11:21:03.926460 5640 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:03.929634 5647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:03.935196 5647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:03.943344 5647 catalog_manager.cc:1347] Generated new cluster ID: 2a840afeb35b4657aa75339d8bdfb3aa
I20260929 11:21:03.943445 5647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:03.960142 5647 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:03.961447 5647 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:03.971758 5647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07975b45adf410daeec96d060815247: Generated new TSK 0
I20260929 11:21:03.972543 5647 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 11:21:04.018764 5664 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 11:21:04.018996 5664 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:21:04.020074 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:03.995713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.97.62:42267)
0929 11:21:03.995894 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:03.995901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:03.996535 (+ 634us) server_negotiation.cc:407] Connection header received
0929 11:21:03.996984 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:03.996989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:03.997088 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:03.997236 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:21:03.997961 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:03.999151 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.001390 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.002033 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.002129 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:04.002918 (+ 789us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:21:04.019391 (+ 16473us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 11:21:04.019758 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 11:21:04.020200 5637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 11:21:04.021497 5663 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:03.995726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42267 (local address 127.0.0.1:43900)
0929 11:21:03.996406 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:03.996420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:03.996685 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:03.997281 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:03.997285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:03.997296 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:21:03.997778 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:03.997808 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:03.999285 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:03.999291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:04.001210 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:04.001229 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.002261 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.002343 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:04.002545 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:04.002622 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:21:04.019676 (+ 17054us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 11:21:04.020010 (+ 334us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":384,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 11:21:04.023037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5568
I20260929 11:21:04.024392 5629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:04.114303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5568
2026-09-29T11:21:04Z chronyd exiting
I20260929 11:21:04.163862 388 test_util.cc:175] -----------------------------------------------
I20260929 11:21:04.164060 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790680821855275-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 11:21:04.167608 388 mini_oidc.cc:152] Starting JWKS server
I20260929 11:21:04.169427 388 webserver.cc:460] Webserver started at https://127.0.0.1:45291/ using document root <none> and password file <none>
I20260929 11:21:04.169669 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:21:04.170315 388 webserver.cc:460] Webserver started at http://127.0.0.1:41271/ using document root <none> and password file <none>
2026-09-29T11:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:04Z Disabled control of system clock
I20260929 11:21:04.194959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40719
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40719
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45291/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/testchainca.cert with env {}
W20260929 11:21:04.500701 5671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:04.500982 5671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:04.501067 5671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:04.507741 5671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:04.507833 5671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:04.507866 5671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:04.507916 5671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:04.507951 5671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:21:04.508011 5671 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:21:04.508080 5671 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45291/jwks.json
I20260929 11:21:04.517577 5671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45291/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:04.518765 5671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:04.520015 5671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:04.531227 5676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:04.531275 5677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:04.531188 5679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:04.532492 5671 server_base.cc:1034] running on GCE node
I20260929 11:21:04.533282 5671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:04.534580 5671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:04.535771 5671 hybrid_clock.cc:648] HybridClock initialized: now 1790680864535739 us; error 29 us; skew 500 ppm
I20260929 11:21:04.536096 5671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:04.538245 5671 webserver.cc:460] Webserver started at http://127.0.97.62:40407/ using document root <none> and password file <none>
I20260929 11:21:04.538877 5671 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:04.539000 5671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:04.539312 5671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:04.541975 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "6e863f2a6fa54401aaece5f68918646b"
format_stamp: "Formatted at 2026-09-29 11:21:04 on dist-test-slave-srst"
server_key: "4a96f6f8c29f2fd082dc18f523fa20d9"
server_key_iv: "6800f0447a1a8e1d2727a336afab88e0"
server_key_version: "encryptionkey@0"
I20260929 11:21:04.542704 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e863f2a6fa54401aaece5f68918646b"
format_stamp: "Formatted at 2026-09-29 11:21:04 on dist-test-slave-srst"
server_key: "4a96f6f8c29f2fd082dc18f523fa20d9"
server_key_iv: "6800f0447a1a8e1d2727a336afab88e0"
server_key_version: "encryptionkey@0"
I20260929 11:21:04.547322 5671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 11:21:04.550535 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:04.551735 5671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 11:21:04.551908 5671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "6e863f2a6fa54401aaece5f68918646b"
format_stamp: "Formatted at 2026-09-29 11:21:04 on dist-test-slave-srst"
server_key: "4a96f6f8c29f2fd082dc18f523fa20d9"
server_key_iv: "6800f0447a1a8e1d2727a336afab88e0"
server_key_version: "encryptionkey@0"
I20260929 11:21:04.552099 5671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:04.571097 5671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:04.571856 5671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:04.572120 5671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:04.589808 5671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40719
I20260929 11:21:04.589833 5736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40719 every 8 connection(s)
I20260929 11:21:04.591030 5671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:04.594633 5737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:04.599987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5671
I20260929 11:21:04.600167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:04.600910 388 external_mini_cluster.cc:1428] Setting key 60bcdcd2e8b505faa8f632df09d00af3
I20260929 11:21:04.605716 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b: Bootstrap starting.
I20260929 11:21:04.608749 5737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:04.609678 5737 log.cc:824] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:04.612797 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b: No bootstrap required, opened a new log
I20260929 11:21:04.613656 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:04.602796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.0.97.62:40719)
0929 11:21:04.603776 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:04.603786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:04.603825 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:21:04.605510 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:04.605531 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:04.606005 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:04.606350 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:04.606860 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.608051 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.609425 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.610231 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.610417 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:04.611474 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:04.611483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:04.611495 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:04.612000 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:04.612070 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:04.612088 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:04.612170 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:04.612419 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 11:21:04.612726 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":359,"threads_started":1}
I20260929 11:21:04.617286 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:21:04.621918 5737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e863f2a6fa54401aaece5f68918646b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40719 } }
I20260929 11:21:04.622208 5737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:04.622345 5737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e863f2a6fa54401aaece5f68918646b, State: Initialized, Role: FOLLOWER
I20260929 11:21:04.622807 5737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e863f2a6fa54401aaece5f68918646b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40719 } }
I20260929 11:21:04.622995 5737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:04.623088 5737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:04.623204 5737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:04.625643 5737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e863f2a6fa54401aaece5f68918646b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40719 } }
I20260929 11:21:04.626041 5737 leader_election.cc:302] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e863f2a6fa54401aaece5f68918646b; no voters:
I20260929 11:21:04.626533 5737 leader_election.cc:288] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:04.626672 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:04.627339 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [term 1 LEADER]: Becoming Leader. State: Replica: 6e863f2a6fa54401aaece5f68918646b, State: Running, Role: LEADER
I20260929 11:21:04.627930 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e863f2a6fa54401aaece5f68918646b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40719 } }
I20260929 11:21:04.628682 5737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:04.630548 5744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e863f2a6fa54401aaece5f68918646b. Latest consensus state: current_term: 1 leader_uuid: "6e863f2a6fa54401aaece5f68918646b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e863f2a6fa54401aaece5f68918646b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40719 } } }
I20260929 11:21:04.631042 5744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:04.631001 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e863f2a6fa54401aaece5f68918646b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e863f2a6fa54401aaece5f68918646b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40719 } } }
I20260929 11:21:04.631268 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:04.632997 5750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:04.637279 5750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:04.645007 5750 catalog_manager.cc:1347] Generated new cluster ID: 1915b6c13b82435889fc64840b6c97d1
I20260929 11:21:04.645102 5750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:04.674045 5750 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:04.675971 5750 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:04.689549 5750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e863f2a6fa54401aaece5f68918646b: Generated new TSK 0
I20260929 11:21:04.690536 5750 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 11:21:04.724223 5767 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 11:21:04.724341 5767 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:21:04.725536 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:04.706165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.97.62:40719)
0929 11:21:04.706361 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:04.706371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:04.706898 (+ 527us) server_negotiation.cc:407] Connection header received
0929 11:21:04.707126 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:04.707130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:04.707190 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:04.707301 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:21:04.707843 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.708644 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.710552 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.711012 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.711096 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:04.711636 (+ 540us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:21:04.724851 (+ 13215us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 11:21:04.725197 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 11:21:04.725731 5766 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:04.706139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40719 (local address 127.0.0.1:57356)
0929 11:21:04.706788 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:04.706811 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:04.706957 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:04.707342 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:04.707347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:04.707362 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:21:04.707670 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:04.707681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.708785 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.708790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:04.710419 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:04.710433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:04.711125 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:04.711168 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:04.711361 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:04.711402 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:21:04.725061 (+ 13659us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 11:21:04.725319 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":343,"threads_started":1}
W20260929 11:21:04.725942 5740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 11:21:04.726878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5671
I20260929 11:21:04.727885 5732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:04.819572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5671
2026-09-29T11:21:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (708 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 11:21:04.876339 388 mini_oidc.cc:152] Starting JWKS server
I20260929 11:21:04.878481 388 webserver.cc:460] Webserver started at https://127.0.0.1:37557/ using document root <none> and password file <none>
I20260929 11:21:04.878762 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:21:04.879453 388 webserver.cc:460] Webserver started at http://127.0.0.1:44343/ using document root <none> and password file <none>
2026-09-29T11:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:04Z Disabled control of system clock
I20260929 11:21:04.904726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34701
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34701
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 11:21:05.223520 5774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:05.223814 5774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:05.223997 5774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:05.231220 5774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:05.231331 5774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:05.231451 5774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:05.231575 5774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:05.231662 5774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:21:05.231761 5774 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:21:05.231842 5774 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37557/jwks.json
I20260929 11:21:05.241146 5774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:05.242381 5774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:05.244067 5774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:05.255637 5780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:05.255705 5782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:05.255981 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:05.256649 5774 server_base.cc:1034] running on GCE node
I20260929 11:21:05.257354 5774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:05.258635 5774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:05.259852 5774 hybrid_clock.cc:648] HybridClock initialized: now 1790680865259823 us; error 41 us; skew 500 ppm
I20260929 11:21:05.260180 5774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:05.267455 5774 webserver.cc:460] Webserver started at http://127.0.97.62:34603/ using document root <none> and password file <none>
I20260929 11:21:05.268157 5774 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:05.268285 5774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:05.268684 5774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:05.271490 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "bc99c83d6d934dfda11ffb06a43f4534"
format_stamp: "Formatted at 2026-09-29 11:21:05 on dist-test-slave-srst"
server_key: "075e8ac01d6ecf449a62a19a68bba829"
server_key_iv: "fca238211419ffd5360262d13f904f65"
server_key_version: "encryptionkey@0"
I20260929 11:21:05.272279 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc99c83d6d934dfda11ffb06a43f4534"
format_stamp: "Formatted at 2026-09-29 11:21:05 on dist-test-slave-srst"
server_key: "075e8ac01d6ecf449a62a19a68bba829"
server_key_iv: "fca238211419ffd5360262d13f904f65"
server_key_version: "encryptionkey@0"
I20260929 11:21:05.277321 5774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 11:21:05.280578 5788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:05.281955 5774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:21:05.282153 5774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "bc99c83d6d934dfda11ffb06a43f4534"
format_stamp: "Formatted at 2026-09-29 11:21:05 on dist-test-slave-srst"
server_key: "075e8ac01d6ecf449a62a19a68bba829"
server_key_iv: "fca238211419ffd5360262d13f904f65"
server_key_version: "encryptionkey@0"
I20260929 11:21:05.282357 5774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:05.308969 5774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:05.309839 5774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:05.310113 5774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:05.327528 5774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34701
I20260929 11:21:05.327548 5839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34701 every 8 connection(s)
I20260929 11:21:05.328814 5774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:05.331715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5774
I20260929 11:21:05.331919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:05.332731 388 external_mini_cluster.cc:1428] Setting key 2d74a0ea3744e56eb0488bb042918203
I20260929 11:21:05.332826 5840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:05.345950 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:05.334632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.97.62:34701)
0929 11:21:05.335601 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:05.335611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:05.335653 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:21:05.337312 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:05.337337 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:05.337783 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:05.338123 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:05.338610 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:05.339924 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:05.341671 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:05.342393 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:05.342545 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:05.343718 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:05.343725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:05.343740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:05.344292 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:05.344363 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:05.344383 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:05.344465 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:05.344737 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:21:05.345067 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":379,"threads_started":1}
I20260929 11:21:05.346115 5840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534: Bootstrap starting.
I20260929 11:21:05.349311 5840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:05.349610 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:21:05.350524 5840 log.cc:824] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:05.353392 5840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534: No bootstrap required, opened a new log
I20260929 11:21:05.360002 5840 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc99c83d6d934dfda11ffb06a43f4534" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34701 } }
I20260929 11:21:05.360314 5840 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:05.360381 5840 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc99c83d6d934dfda11ffb06a43f4534, State: Initialized, Role: FOLLOWER
I20260929 11:21:05.360895 5840 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc99c83d6d934dfda11ffb06a43f4534" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34701 } }
I20260929 11:21:05.361131 5840 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:05.361282 5840 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:05.361442 5840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:05.363386 5840 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc99c83d6d934dfda11ffb06a43f4534" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34701 } }
I20260929 11:21:05.363806 5840 leader_election.cc:302] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc99c83d6d934dfda11ffb06a43f4534; no voters:
I20260929 11:21:05.364377 5840 leader_election.cc:288] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:05.364473 5845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:05.365211 5845 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [term 1 LEADER]: Becoming Leader. State: Replica: bc99c83d6d934dfda11ffb06a43f4534, State: Running, Role: LEADER
I20260929 11:21:05.365733 5845 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc99c83d6d934dfda11ffb06a43f4534" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34701 } }
I20260929 11:21:05.366420 5840 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:05.368368 5846 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc99c83d6d934dfda11ffb06a43f4534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc99c83d6d934dfda11ffb06a43f4534" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34701 } } }
I20260929 11:21:05.368692 5846 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:05.370431 5850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:05.369938 5847 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc99c83d6d934dfda11ffb06a43f4534. Latest consensus state: current_term: 1 leader_uuid: "bc99c83d6d934dfda11ffb06a43f4534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc99c83d6d934dfda11ffb06a43f4534" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34701 } } }
I20260929 11:21:05.371411 5847 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:05.375439 5850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:05.383498 5850 catalog_manager.cc:1347] Generated new cluster ID: 0fc48afaf3794fbc84f0a2e0b4dbaf9d
I20260929 11:21:05.383630 5850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:05.416402 5850 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:05.417747 5850 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:05.426649 5850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc99c83d6d934dfda11ffb06a43f4534: Generated new TSK 0
I20260929 11:21:05.427500 5850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:21:05.454622 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:05.445784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.0.97.62:34701)
0929 11:21:05.445972 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:05.445979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:05.446434 (+ 455us) server_negotiation.cc:407] Connection header received
0929 11:21:05.446642 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:05.446649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:05.446749 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:05.446918 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:21:05.447362 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:05.448476 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:05.450328 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:05.451035 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:05.451158 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:05.451646 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:21:05.453950 (+ 2304us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37557/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 11:21:05.454292 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37557/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 11:21:05.454752 5843 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37557/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 11:21:05.456203 5869 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:05.445676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34701 (local address 127.0.0.1:52288)
0929 11:21:05.446336 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:05.446351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:05.446467 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:05.446942 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:05.446946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:05.446955 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:21:05.447226 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:05.447234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:05.448706 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:05.448710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:05.450160 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:05.450172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:05.451156 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:05.451220 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:05.451339 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:05.451371 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:21:05.454164 (+ 2793us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37557/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 11:21:05.455841 (+ 1677us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37557/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":1395,"thread_start_us":367,"threads_started":1}
I20260929 11:21:05.457130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5774
I20260929 11:21:05.458235 5835 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 11:21:05.470299 5870 webserver.cc:533] Webserver: sslize failed:
E20260929 11:21:05.470434 5870 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:21:05.551573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5774
2026-09-29T11:21:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T11:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:05Z Disabled control of system clock
I20260929 11:21:05.630947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33871
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33871
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/testchainca.cert with env {}
W20260929 11:21:05.945369 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:05.945667 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:05.945751 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:05.953027 5875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:05.953145 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:05.953181 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:05.953240 5875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:05.953284 5875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:21:05.953344 5875 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:21:05.953382 5875 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 11:21:05.963049 5875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:05.964300 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:05.965598 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:05.977650 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:05.977650 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:05.977764 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:05.978322 5875 server_base.cc:1034] running on GCE node
I20260929 11:21:05.979144 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:05.980449 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:05.981648 5875 hybrid_clock.cc:648] HybridClock initialized: now 1790680865981607 us; error 59 us; skew 500 ppm
I20260929 11:21:05.981978 5875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:05.984123 5875 webserver.cc:460] Webserver started at http://127.0.97.62:39911/ using document root <none> and password file <none>
I20260929 11:21:05.984874 5875 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:05.984989 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:05.985314 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:05.988121 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "858eb7335e0c4f1f850a24486e12d89b"
format_stamp: "Formatted at 2026-09-29 11:21:05 on dist-test-slave-srst"
server_key: "56dc688ce817d6eeae18585caad7f79e"
server_key_iv: "431e82a9da7aa2ca93fa1ba4a33c94a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:05.988921 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "858eb7335e0c4f1f850a24486e12d89b"
format_stamp: "Formatted at 2026-09-29 11:21:05 on dist-test-slave-srst"
server_key: "56dc688ce817d6eeae18585caad7f79e"
server_key_iv: "431e82a9da7aa2ca93fa1ba4a33c94a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:05.993811 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:21:05.997406 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:05.999205 5875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:05.999362 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "858eb7335e0c4f1f850a24486e12d89b"
format_stamp: "Formatted at 2026-09-29 11:21:05 on dist-test-slave-srst"
server_key: "56dc688ce817d6eeae18585caad7f79e"
server_key_iv: "431e82a9da7aa2ca93fa1ba4a33c94a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:05.999567 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:06.025437 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:06.026278 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:06.026567 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:06.044018 5875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33871
I20260929 11:21:06.044037 5940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33871 every 8 connection(s)
I20260929 11:21:06.045418 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:06.048868 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5875
I20260929 11:21:06.048951 5941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:06.049054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:06.049723 388 external_mini_cluster.cc:1428] Setting key 7cf642a6c23dfcc48432727680fdddb4
I20260929 11:21:06.063212 5941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b: Bootstrap starting.
I20260929 11:21:06.065505 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:06.051650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.0.97.62:33871)
0929 11:21:06.052755 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:06.052773 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:06.052826 (+ 53us) server_negotiation.cc:407] Connection header received
0929 11:21:06.055473 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:06.055501 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:06.056102 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:06.056550 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:06.057100 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:06.058959 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:06.060231 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:06.061333 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:06.061498 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:06.062626 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:06.062636 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:06.062654 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:06.063361 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:06.063469 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:06.063497 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:06.063605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:06.063829 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 11:21:06.064141 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":351,"threads_started":1}
I20260929 11:21:06.067067 5941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:06.068291 5941 log.cc:824] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:06.069434 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:21:06.071098 5941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b: No bootstrap required, opened a new log
I20260929 11:21:06.079126 5941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858eb7335e0c4f1f850a24486e12d89b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33871 } }
I20260929 11:21:06.079489 5941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:06.079609 5941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858eb7335e0c4f1f850a24486e12d89b, State: Initialized, Role: FOLLOWER
I20260929 11:21:06.080227 5941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858eb7335e0c4f1f850a24486e12d89b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33871 } }
I20260929 11:21:06.080456 5941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:06.080617 5941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:06.080798 5941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:06.082726 5941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858eb7335e0c4f1f850a24486e12d89b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33871 } }
I20260929 11:21:06.083214 5941 leader_election.cc:302] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 858eb7335e0c4f1f850a24486e12d89b; no voters:
I20260929 11:21:06.083796 5941 leader_election.cc:288] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:06.083981 5946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:06.084792 5946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [term 1 LEADER]: Becoming Leader. State: Replica: 858eb7335e0c4f1f850a24486e12d89b, State: Running, Role: LEADER
I20260929 11:21:06.085321 5946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858eb7335e0c4f1f850a24486e12d89b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33871 } }
I20260929 11:21:06.085930 5941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:06.088104 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "858eb7335e0c4f1f850a24486e12d89b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858eb7335e0c4f1f850a24486e12d89b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33871 } } }
I20260929 11:21:06.088148 5948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 858eb7335e0c4f1f850a24486e12d89b. Latest consensus state: current_term: 1 leader_uuid: "858eb7335e0c4f1f850a24486e12d89b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858eb7335e0c4f1f850a24486e12d89b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33871 } } }
I20260929 11:21:06.088503 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:06.088650 5948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:06.090493 5954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:06.095016 5954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:06.104934 5954 catalog_manager.cc:1347] Generated new cluster ID: 4cfd1503e2814ab4a93754d2f49a5726
I20260929 11:21:06.105031 5954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:06.118573 5954 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:06.119834 5954 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:06.130270 5954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 858eb7335e0c4f1f850a24486e12d89b: Generated new TSK 0
I20260929 11:21:06.131081 5954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:21:06.399881 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:06.145665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.0.97.62:33871)
0929 11:21:06.145873 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:06.145888 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:06.146256 (+ 368us) server_negotiation.cc:407] Connection header received
0929 11:21:06.146493 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:06.146497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:06.146559 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:06.146756 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:21:06.147217 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:06.147940 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:06.149375 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:06.149838 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:06.149977 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:06.150408 (+ 431us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:21:06.399180 (+248772us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 11:21:06.399582 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 11:21:06.400121 5944 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 11:21:06.400174 5970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:06.145599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33871 (local address 127.0.0.1:38364)
0929 11:21:06.146144 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:06.146159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:06.146330 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:06.146778 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:06.146783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:06.146796 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:21:06.147073 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:06.147081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:06.148090 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:06.148102 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:06.149227 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:06.149238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:06.149958 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:06.150008 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:06.150140 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:06.150171 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:21:06.399463 (+249292us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 11:21:06.399749 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33871: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":316,"threads_started":1}
I20260929 11:21:06.401968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5875
I20260929 11:21:06.403098 5936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:06.512023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5875
2026-09-29T11:21:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (942 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (401 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (400 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:07 dist-test-slave-srst krb5kdc[5985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:07 dist-test-slave-srst krb5kdc[5985](info): set up 2 sockets
Sep 29 11:21:07 dist-test-slave-srst krb5kdc[5985](info): commencing operation
krb5kdc: starting...
W20260929 11:21:08.393901 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:08 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:21:08.599915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:08.910410 5998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:08.910714 5998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:08.910799 5998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:08.917470 5998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:08.917564 5998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:08.917605 5998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:08.917656 5998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:08.917695 5998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:08.926779 5998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:08.927964 5998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:08.929219 5998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:08.940644 6004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:08.940766 6003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:08.940745 6006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:08.941357 5998 server_base.cc:1034] running on GCE node
I20260929 11:21:08.942078 5998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:08.943326 5998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:08.944537 5998 hybrid_clock.cc:648] HybridClock initialized: now 1790680868944498 us; error 52 us; skew 500 ppm
I20260929 11:21:08.944890 5998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:08 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:08.947748 5998 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:21:08.949149 5998 webserver.cc:460] Webserver started at http://127.0.97.62:39673/ using document root <none> and password file <none>
I20260929 11:21:08.949795 5998 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:08.949931 5998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:08.950233 5998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:08.953186 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "d9d7af8dc67e489792756e1d169ea86d"
format_stamp: "Formatted at 2026-09-29 11:21:08 on dist-test-slave-srst"
server_key: "dce0c0f7a33afff042626aef4db5239c"
server_key_iv: "7e9aafe70caa2d5d594487ca36c0b157"
server_key_version: "encryptionkey@0"
I20260929 11:21:08.953945 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "d9d7af8dc67e489792756e1d169ea86d"
format_stamp: "Formatted at 2026-09-29 11:21:08 on dist-test-slave-srst"
server_key: "dce0c0f7a33afff042626aef4db5239c"
server_key_iv: "7e9aafe70caa2d5d594487ca36c0b157"
server_key_version: "encryptionkey@0"
I20260929 11:21:08.958909 5998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 11:21:08.962193 6013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:08.963388 5998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:21:08.963578 5998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "d9d7af8dc67e489792756e1d169ea86d"
format_stamp: "Formatted at 2026-09-29 11:21:08 on dist-test-slave-srst"
server_key: "dce0c0f7a33afff042626aef4db5239c"
server_key_iv: "7e9aafe70caa2d5d594487ca36c0b157"
server_key_version: "encryptionkey@0"
I20260929 11:21:08.963764 5998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:08.985162 5998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:08.989215 5998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:08.989513 5998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:09.007138 5998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35731
I20260929 11:21:09.007174 6064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35731 every 8 connection(s)
I20260929 11:21:09.008441 5998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:09.012089 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:09.018697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 5998
I20260929 11:21:09.018958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:09.019824 388 external_mini_cluster.cc:1428] Setting key f6caeadd8910d5da684840c5679f09b6
I20260929 11:21:09.024295 6065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d: Bootstrap starting.
I20260929 11:21:09.028244 6065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:09.029626 6065 log.cc:824] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:09.032935 6065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d: No bootstrap required, opened a new log
Sep 29 11:21:09 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:09.043977 6065 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d7af8dc67e489792756e1d169ea86d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } }
I20260929 11:21:09.044422 6065 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:09.044545 6065 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d7af8dc67e489792756e1d169ea86d, State: Initialized, Role: FOLLOWER
I20260929 11:21:09.045047 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:09.022007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.97.62:35731)
0929 11:21:09.022802 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:09.022813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:09.022872 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:21:09.024537 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:09.024557 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:09.024989 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:09.025292 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:09.027262 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:09.028575 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:09.030489 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:09.031321 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:09.031453 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:09.036208 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:09.036224 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:09.036245 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:09.036298 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:09.040540 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:09.041190 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:09.041205 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:09.041214 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:09.041449 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:09.041783 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:09.041789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:09.041793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:09.042551 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:09.042851 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:09.043155 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 11:21:09.043616 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":359,"threads_started":1}
I20260929 11:21:09.045351 6065 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d7af8dc67e489792756e1d169ea86d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } }
I20260929 11:21:09.045594 6065 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:09.045753 6065 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:09.045938 6065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:09.048909 6065 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d7af8dc67e489792756e1d169ea86d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } }
I20260929 11:21:09.049374 6065 leader_election.cc:302] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9d7af8dc67e489792756e1d169ea86d; no voters:
I20260929 11:21:09.049938 6065 leader_election.cc:288] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:09.050065 6070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:09.050777 6070 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [term 1 LEADER]: Becoming Leader. State: Replica: d9d7af8dc67e489792756e1d169ea86d, State: Running, Role: LEADER
I20260929 11:21:09.051275 6070 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d7af8dc67e489792756e1d169ea86d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } }
I20260929 11:21:09.052000 6065 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:09.054291 6072 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9d7af8dc67e489792756e1d169ea86d. Latest consensus state: current_term: 1 leader_uuid: "d9d7af8dc67e489792756e1d169ea86d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d7af8dc67e489792756e1d169ea86d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } }
I20260929 11:21:09.054759 6072 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:09.055399 6071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9d7af8dc67e489792756e1d169ea86d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d7af8dc67e489792756e1d169ea86d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35731 } } }
I20260929 11:21:09.055816 6071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:09.056367 6079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:09.061480 6079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:09.069947 6079 catalog_manager.cc:1347] Generated new cluster ID: 904be9794eac42ec9557f3149ffd6ff4
I20260929 11:21:09.070055 6079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:09.082939 6079 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:09.084167 6079 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:09.090747 6079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9d7af8dc67e489792756e1d169ea86d: Generated new TSK 0
I20260929 11:21:09.091549 6079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:21:09.146250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35731
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:09.483741 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:09.484011 6091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:09.484166 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:09.491405 6091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:09.491511 6091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:09.491724 6091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:09.501025 6091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:09.502228 6091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:09.503525 6091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:09.516080 6096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:09.516129 6099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:09.516158 6097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:09.516985 6091 server_base.cc:1034] running on GCE node
I20260929 11:21:09.517643 6091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:09.518538 6091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:09.519750 6091 hybrid_clock.cc:648] HybridClock initialized: now 1790680869519691 us; error 29 us; skew 500 ppm
I20260929 11:21:09.520077 6091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:09 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:09.523135 6091 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:21:09.524562 6091 webserver.cc:460] Webserver started at http://127.0.97.1:42543/ using document root <none> and password file <none>
I20260929 11:21:09.525305 6091 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:09.525446 6091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:09.525753 6091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:09.528517 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "ed1d662621c941809435dbd5a765f723"
format_stamp: "Formatted at 2026-09-29 11:21:09 on dist-test-slave-srst"
server_key: "562d57078a0492fad0cb7d985a35bada"
server_key_iv: "86c697277068229a0be6b9a5f928c125"
server_key_version: "encryptionkey@0"
I20260929 11:21:09.529314 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ed1d662621c941809435dbd5a765f723"
format_stamp: "Formatted at 2026-09-29 11:21:09 on dist-test-slave-srst"
server_key: "562d57078a0492fad0cb7d985a35bada"
server_key_iv: "86c697277068229a0be6b9a5f928c125"
server_key_version: "encryptionkey@0"
I20260929 11:21:09.534219 6091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:21:09.537773 6106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:09.539002 6091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:09.539186 6091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "ed1d662621c941809435dbd5a765f723"
format_stamp: "Formatted at 2026-09-29 11:21:09 on dist-test-slave-srst"
server_key: "562d57078a0492fad0cb7d985a35bada"
server_key_iv: "86c697277068229a0be6b9a5f928c125"
server_key_version: "encryptionkey@0"
I20260929 11:21:09.539387 6091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:09.560918 6091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:09.564774 6091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:09.565079 6091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:09.565990 6091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:09.567304 6091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:09.567410 6091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:09.567552 6091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:09.567622 6091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:09.600178 6091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46257
I20260929 11:21:09.600231 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46257 every 8 connection(s)
I20260929 11:21:09.601481 6091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:09.608191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6091
I20260929 11:21:09.608423 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:09.609122 388 external_mini_cluster.cc:1428] Setting key 7c077d2da02eb8d0fae157b2701f90f0
Sep 29 11:21:09 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:09.624814 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:09.604191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41797 (local address 127.0.97.62:35731)
0929 11:21:09.604724 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:09.604729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:09.605720 (+ 991us) server_negotiation.cc:407] Connection header received
0929 11:21:09.607544 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:09.607549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:09.607639 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:09.607766 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:09.609508 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:09.610481 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:09.612895 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:09.613269 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:09.613369 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:09.617892 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:09.617897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:09.617900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:09.617963 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:09.621486 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:09.622244 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:09.622252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:09.622259 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:09.622401 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:09.622811 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:09.622819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:09.622825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:09.623232 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:09.623380 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:09.624051 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 11:21:09.624292 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":351,"thread_start_us":210,"threads_started":1}
I20260929 11:21:09.625480 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:09.604826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35731 (local address 127.0.97.1:41797)
0929 11:21:09.605544 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:09.605587 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:09.607278 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:09.608002 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:09.608010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:09.608706 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:09.609343 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:09.609363 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:09.610636 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:09.610640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:09.612743 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:09.612762 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:09.613507 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:09.613517 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:09.613852 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:09.614600 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:09.614641 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:09.617620 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:09.621658 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:09.621665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:09.621678 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:09.622103 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:09.622502 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:09.622505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:09.622508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:09.622698 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:09.623348 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:09.623354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:09.623679 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 11:21:09.623900 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 11:21:09.624285 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":295,"threads_started":1}
I20260929 11:21:09.627089 6219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35731
I20260929 11:21:09.627439 6219 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:09.628407 6219 heartbeater.cc:511] Master 127.0.97.62:35731 requested a full tablet report, sending...
I20260929 11:21:09.630666 6030 ts_manager.cc:194] Registered new tserver with Master: ed1d662621c941809435dbd5a765f723 (127.0.97.1:46257)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:21:09.632169 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:21:09.656896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35731
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:09.968735 6225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:09.969002 6225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:09.969182 6225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:09.976276 6225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:09.976375 6225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:09.976580 6225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:09.986059 6225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:09.987277 6225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:09.988582 6225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:10.004916 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:10.005275 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:10.006021 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:10.007010 6225 server_base.cc:1034] running on GCE node
I20260929 11:21:10.007673 6225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:10.008857 6225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:10.010090 6225 hybrid_clock.cc:648] HybridClock initialized: now 1790680870010021 us; error 57 us; skew 500 ppm
I20260929 11:21:10.010442 6225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.014432 6225 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:21:10.017828 6225 webserver.cc:460] Webserver started at http://127.0.97.2:33497/ using document root <none> and password file <none>
I20260929 11:21:10.018579 6225 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:10.018781 6225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:10.019202 6225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:10.023015 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "0530dbea99f941a7bd2da76fae5efc4f"
format_stamp: "Formatted at 2026-09-29 11:21:10 on dist-test-slave-srst"
server_key: "b9e23220efaaedadedc977c103d102d3"
server_key_iv: "aae54f1d160180cab887444da0eabfc7"
server_key_version: "encryptionkey@0"
I20260929 11:21:10.023955 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0530dbea99f941a7bd2da76fae5efc4f"
format_stamp: "Formatted at 2026-09-29 11:21:10 on dist-test-slave-srst"
server_key: "b9e23220efaaedadedc977c103d102d3"
server_key_iv: "aae54f1d160180cab887444da0eabfc7"
server_key_version: "encryptionkey@0"
I20260929 11:21:10.030498 6225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260929 11:21:10.035326 6240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:10.036907 6225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:10.037089 6225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "0530dbea99f941a7bd2da76fae5efc4f"
format_stamp: "Formatted at 2026-09-29 11:21:10 on dist-test-slave-srst"
server_key: "b9e23220efaaedadedc977c103d102d3"
server_key_iv: "aae54f1d160180cab887444da0eabfc7"
server_key_version: "encryptionkey@0"
I20260929 11:21:10.037333 6225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:10.054255 6225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:10.058183 6225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:10.058410 6225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:10.059248 6225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:10.060457 6225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:10.060526 6225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:10.060668 6225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:10.060730 6225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 11:21:10.092555 6225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40765
I20260929 11:21:10.092630 6352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40765 every 8 connection(s)
I20260929 11:21:10.093916 6225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:10.096661 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6225
I20260929 11:21:10.096830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:10.097290 388 external_mini_cluster.cc:1428] Setting key 93c8180ac580c787c7e35deb29fb28f9
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:10.113786 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.096437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37149 (local address 127.0.97.62:35731)
0929 11:21:10.096675 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.096681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.097995 (+ 1314us) server_negotiation.cc:407] Connection header received
0929 11:21:10.099811 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.099817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.099901 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.100000 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.101456 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.102167 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.103843 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.104592 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.104750 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.108394 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.108398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.108401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.108429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.110931 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.111641 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.111648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.111655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.111780 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.112182 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.112189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.112192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.112445 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.112601 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.113256 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.113459 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:21:10.114761 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.097044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35731 (local address 127.0.97.2:37149)
0929 11:21:10.097852 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:10.097897 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:10.099554 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:10.100225 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:10.100233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:10.100766 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:10.101298 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:10.101314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.102340 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.102343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:10.103683 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:10.103694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.104811 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.104816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:10.105010 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.105697 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:10.105735 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:10.108191 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:10.111063 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:10.111074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:10.111108 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:10.111525 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:10.111886 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:10.111892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:10.111895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:10.112074 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:10.112575 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:10.112582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:10.112918 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 11:21:10.113175 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 11:21:10.113671 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:21:10.116153 6353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35731
I20260929 11:21:10.116472 6353 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:10.117197 6353 heartbeater.cc:511] Master 127.0.97.62:35731 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 11:21:10.118573 6030 ts_manager.cc:194] Registered new tserver with Master: 0530dbea99f941a7bd2da76fae5efc4f (127.0.97.2:40765)
I20260929 11:21:10.119372 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:21:10.143558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35731
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:10.467203 6358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:10.467489 6358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:10.467641 6358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:10.474874 6358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:10.474977 6358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:10.475159 6358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:10.484747 6358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:10.485992 6358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:10.487325 6358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:10.500567 6363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:10.500626 6364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:10.500649 6366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:10.501544 6358 server_base.cc:1034] running on GCE node
I20260929 11:21:10.502163 6358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:10.503115 6358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:10.504359 6358 hybrid_clock.cc:648] HybridClock initialized: now 1790680870504308 us; error 38 us; skew 500 ppm
I20260929 11:21:10.504798 6358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.511451 6358 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:21:10.512993 6358 webserver.cc:460] Webserver started at http://127.0.97.3:45315/ using document root <none> and password file <none>
I20260929 11:21:10.513659 6358 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:10.513784 6358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:10.514076 6358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:10.517043 6358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "be566fb4762d419e96a420df60b7918f"
format_stamp: "Formatted at 2026-09-29 11:21:10 on dist-test-slave-srst"
server_key: "116d3777ff994534de8e75648002765a"
server_key_iv: "fa03d8b9cb7ebfcd98f3107afcbc37f2"
server_key_version: "encryptionkey@0"
I20260929 11:21:10.517800 6358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "be566fb4762d419e96a420df60b7918f"
format_stamp: "Formatted at 2026-09-29 11:21:10 on dist-test-slave-srst"
server_key: "116d3777ff994534de8e75648002765a"
server_key_iv: "fa03d8b9cb7ebfcd98f3107afcbc37f2"
server_key_version: "encryptionkey@0"
I20260929 11:21:10.522514 6358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:21:10.525719 6373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:10.526948 6358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:10.527118 6358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "be566fb4762d419e96a420df60b7918f"
format_stamp: "Formatted at 2026-09-29 11:21:10 on dist-test-slave-srst"
server_key: "116d3777ff994534de8e75648002765a"
server_key_iv: "fa03d8b9cb7ebfcd98f3107afcbc37f2"
server_key_version: "encryptionkey@0"
I20260929 11:21:10.527323 6358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:10.556391 6358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:10.560335 6358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:10.560701 6358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:10.561707 6358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:10.563069 6358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:10.563184 6358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:10.563331 6358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:10.563410 6358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:10.595242 6358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38499
I20260929 11:21:10.595280 6485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38499 every 8 connection(s)
I20260929 11:21:10.596591 6358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:10.603494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6358
I20260929 11:21:10.603669 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:10.604212 388 external_mini_cluster.cc:1428] Setting key 3b471d5dd5b36f1ef4a45f4eaa285c70
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:10.615669 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.599021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39441 (local address 127.0.97.62:35731)
0929 11:21:10.599180 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.599183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.600505 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 11:21:10.602236 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.602241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.602334 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.602440 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.604387 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.605329 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.606870 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.607225 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.607356 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.610493 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.610497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.610501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.610530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.612882 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.613560 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.613571 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.613574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.613686 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.614062 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.614069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.614072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.614317 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.614456 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.615086 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.615317 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:21:10.616542 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.599589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35731 (local address 127.0.97.3:39441)
0929 11:21:10.600350 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:10.600390 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:10.601983 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:10.602680 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:10.602689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:10.603358 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:10.604178 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:10.604208 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.605478 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.605484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:10.606697 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:10.606708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.607433 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.607442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:10.607668 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.608292 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:10.608313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:10.610305 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:10.613011 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:10.613018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:10.613033 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:10.613444 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:10.613784 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:10.613791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:10.613797 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:10.613962 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:10.614430 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:10.614437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:10.614726 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 11:21:10.614992 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 11:21:10.615494 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":318,"threads_started":1}
I20260929 11:21:10.617799 6486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35731
I20260929 11:21:10.618070 6486 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:10.618811 6486 heartbeater.cc:511] Master 127.0.97.62:35731 requested a full tablet report, sending...
I20260929 11:21:10.620067 6030 ts_manager.cc:194] Registered new tserver with Master: be566fb4762d419e96a420df60b7918f (127.0.97.3:38499)
I20260929 11:21:10.620855 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39441
I20260929 11:21:10.631160 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:21:10.635316 6219 heartbeater.cc:503] Master 127.0.97.62:35731 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/test-admin.keytab.
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.658453 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 11:21:10.663304 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:10.661091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.0.97.1:46257)
0929 11:21:10.661686 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.661694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.661719 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:21:10.661779 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.661786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.661998 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.662122 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.662846 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33794: BlockingRecv error: recv got EOF from 127.0.0.1:33794 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":313,"threads_started":1}
W20260929 11:21:10.663641 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc)
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.665196 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.666553 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:21:10.681303 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.668141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.0.97.1:46257)
0929 11:21:10.668396 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.668402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.668634 (+ 232us) server_negotiation.cc:407] Connection header received
0929 11:21:10.668785 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.668789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.668848 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.668928 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.669813 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.670731 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.672038 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.673019 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.673115 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.676541 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.676551 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.676565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.676638 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.679070 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.679628 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.679634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.679636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.679738 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.680021 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.680024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.680026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.680347 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.680533 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.680875 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.681051 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260929 11:21:10.686852 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:10.685044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.0.97.1:46257)
0929 11:21:10.685247 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.685253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.685481 (+ 228us) server_negotiation.cc:407] Connection header received
0929 11:21:10.685660 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.685665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.685725 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.685829 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.686596 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33814: BlockingRecv error: recv got EOF from 127.0.0.1:33814 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 11:21:10.687016 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.688431 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.689664 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:21:10.702405 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.690729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.0.97.1:46257)
0929 11:21:10.690869 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.690872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.691207 (+ 335us) server_negotiation.cc:407] Connection header received
0929 11:21:10.691334 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.691337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.691402 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.691486 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.692305 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.693058 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.694236 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.694852 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.694954 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.698096 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.698100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.698103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.698131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.700565 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.701103 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.701106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.701108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.701222 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.701505 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.701508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.701509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.701721 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.701895 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.702061 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.702191 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 11:21:10.706156 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:10.704797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.0.97.1:46257)
0929 11:21:10.704958 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.704964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.705081 (+ 117us) server_negotiation.cc:407] Connection header received
0929 11:21:10.705210 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.705213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.705271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.705352 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.705965 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33838: BlockingRecv error: recv got EOF from 127.0.0.1:33838 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 11:21:10.706300 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790680821855275-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.707777 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:10.709014 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:21:10.721817 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.710097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.0.97.1:46257)
0929 11:21:10.710296 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.710301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.710527 (+ 226us) server_negotiation.cc:407] Connection header received
0929 11:21:10.710623 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.710626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.710671 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.710738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.711552 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.712238 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.713531 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.713931 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.714051 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.717197 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.717202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.717205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.717229 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.719387 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.719977 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.719988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.719990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.720121 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.720457 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.720462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.720465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.720752 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.721018 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.721266 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.721463 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:21:10.723500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6091
I20260929 11:21:10.734922 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.724233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.0.97.1:46257)
0929 11:21:10.724412 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.724415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.724844 (+ 429us) server_negotiation.cc:407] Connection header received
0929 11:21:10.724964 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.724967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.725012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.725081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.725954 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.726586 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.727687 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.728059 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.728140 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.729969 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.729973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.729976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.729999 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.732361 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.732969 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.732972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.732974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.733194 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.733594 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.733599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.733602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.733930 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.734162 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.734446 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.734629 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:21:10.735462 6214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:10.816599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6091
I20260929 11:21:10.833253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6225
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:21:10.848548 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.834209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.0.97.2:40765)
0929 11:21:10.834688 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.834695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.834718 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:21:10.834781 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.834788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.835007 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.835188 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.836190 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.837275 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.838491 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.839075 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.839238 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.843287 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.843298 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.843315 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.843349 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.846010 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.846704 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.846710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.846713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.846833 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.847199 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.847202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.847204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.847510 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.847729 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.848082 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.848269 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260929 11:21:10.849939 6348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:10.930250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6225
I20260929 11:21:10.945860 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6358
Sep 29 11:21:10 dist-test-slave-srst krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:21:10.961027 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:10.946961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.0.97.3:38499)
0929 11:21:10.947579 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:10.947586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:10.947614 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:21:10.947691 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:10.947697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:10.947907 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:10.948033 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:10.949108 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.950083 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.951194 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:10.951829 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:10.951970 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:10.956009 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:10.956019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:10.956034 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:10.956063 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:10.958631 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.959173 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.959179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.959181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.959282 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:10.959611 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:10.959614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:10.959616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:10.959913 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:10.960137 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:10.960359 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 11:21:10.960664 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":340,"threads_started":1}
I20260929 11:21:10.962337 6481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:11.043637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6358
I20260929 11:21:11.059142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 5998
I20260929 11:21:11.060487 6060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:11.160161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 5998
2026-09-29T11:21:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3850 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:11 dist-test-slave-srst krb5kdc[6513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:11 dist-test-slave-srst krb5kdc[6513](info): set up 2 sockets
Sep 29 11:21:11 dist-test-slave-srst krb5kdc[6513](info): commencing operation
krb5kdc: starting...
W20260929 11:21:12.245384 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:12 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 11:21:12.445319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33079
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:12.757421 6526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:12.757774 6526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:12.757957 6526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:12.765728 6526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:12.765867 6526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:12.765981 6526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:12.766085 6526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:12.766152 6526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:12.775784 6526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:12.777187 6526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:12.778473 6526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:12.789752 6531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:12.789659 6534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:12.789713 6532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:12.790843 6526 server_base.cc:1034] running on GCE node
I20260929 11:21:12.791558 6526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:12.792991 6526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:12.794214 6526 hybrid_clock.cc:648] HybridClock initialized: now 1790680872794175 us; error 50 us; skew 500 ppm
I20260929 11:21:12.794533 6526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:12 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:12.797592 6526 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 11:21:12.799039 6526 webserver.cc:460] Webserver started at http://127.0.97.62:42515/ using document root <none> and password file <none>
I20260929 11:21:12.799711 6526 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:12.799849 6526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:12.800163 6526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:12.803000 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "08ec214716054f739cd17e77b4e5473d"
format_stamp: "Formatted at 2026-09-29 11:21:12 on dist-test-slave-srst"
server_key: "8178f0f00db1d56c73cacee643de4982"
server_key_iv: "f2a570f2388708b4f1cf50e0500ad55f"
server_key_version: "encryptionkey@0"
I20260929 11:21:12.803731 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "08ec214716054f739cd17e77b4e5473d"
format_stamp: "Formatted at 2026-09-29 11:21:12 on dist-test-slave-srst"
server_key: "8178f0f00db1d56c73cacee643de4982"
server_key_iv: "f2a570f2388708b4f1cf50e0500ad55f"
server_key_version: "encryptionkey@0"
I20260929 11:21:12.808477 6526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 11:21:12.811683 6541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:12.812980 6526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:12.813164 6526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "08ec214716054f739cd17e77b4e5473d"
format_stamp: "Formatted at 2026-09-29 11:21:12 on dist-test-slave-srst"
server_key: "8178f0f00db1d56c73cacee643de4982"
server_key_iv: "f2a570f2388708b4f1cf50e0500ad55f"
server_key_version: "encryptionkey@0"
I20260929 11:21:12.813351 6526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:12.830472 6526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:12.836181 6526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:12.836521 6526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:12.858879 6592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33079 every 8 connection(s)
I20260929 11:21:12.858914 6526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33079
I20260929 11:21:12.860930 6526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:12.865130 6593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:12.867559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6526
I20260929 11:21:12.867760 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:12.868368 388 external_mini_cluster.cc:1428] Setting key ab52dada279bff4659e0e4cc69f463a8
I20260929 11:21:12.877384 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Bootstrap starting.
I20260929 11:21:12.881187 6593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:12.882485 6593 log.cc:824] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:21:12 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:12.885586 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: No bootstrap required, opened a new log
I20260929 11:21:12.891752 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:12.870883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.0.97.62:33079)
0929 11:21:12.871635 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:12.871647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:12.871698 (+ 51us) server_negotiation.cc:407] Connection header received
0929 11:21:12.873370 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:12.873391 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:12.873773 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:12.874095 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:12.875909 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:12.877332 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:12.878976 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:12.879637 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:12.879768 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:12.884682 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:12.884690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:12.884705 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:12.884736 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:12.887818 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:12.888581 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:12.888665 (+ 84us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:12.888672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:12.888837 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:12.889219 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:12.889222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:12.889224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:12.889695 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:12.889911 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:12.890352 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 11:21:12.890702 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":337,"threads_started":1}
I20260929 11:21:12.894445 6593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260929 11:21:12.894865 6593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:12.894987 6593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ec214716054f739cd17e77b4e5473d, State: Initialized, Role: FOLLOWER
I20260929 11:21:12.895459 6593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260929 11:21:12.895653 6593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:12.895740 6593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:12.895851 6593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:12.897930 6593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260929 11:21:12.898449 6593 leader_election.cc:302] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ec214716054f739cd17e77b4e5473d; no voters:
I20260929 11:21:12.899194 6593 leader_election.cc:288] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:12.899343 6598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:12.900048 6598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 LEADER]: Becoming Leader. State: Replica: 08ec214716054f739cd17e77b4e5473d, State: Running, Role: LEADER
I20260929 11:21:12.900513 6598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260929 11:21:12.901230 6593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:12.903676 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec214716054f739cd17e77b4e5473d. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } }
I20260929 11:21:12.903769 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } }
I20260929 11:21:12.904078 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:12.904189 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:12.904894 6605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:12.909868 6605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:12.920303 6605 catalog_manager.cc:1347] Generated new cluster ID: 4d275977890e45628abadcad783f75db
I20260929 11:21:12.920528 6605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:12.931164 6605 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:12.932519 6605 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:12.938916 6605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Generated new TSK 0
I20260929 11:21:12.939647 6605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 11:21:12.988384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:13.304383 6619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:13.304656 6619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:13.304800 6619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:13.311638 6619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:13.311793 6619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:13.311996 6619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:13.321499 6619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:13.322736 6619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:13.323976 6619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:13.336920 6627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:13.336889 6625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:13.337343 6624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:13.337746 6619 server_base.cc:1034] running on GCE node
I20260929 11:21:13.338424 6619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:13.339324 6619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:13.340536 6619 hybrid_clock.cc:648] HybridClock initialized: now 1790680873340487 us; error 68 us; skew 500 ppm
I20260929 11:21:13.340873 6619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:13 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:13.343662 6619 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 11:21:13.345085 6619 webserver.cc:460] Webserver started at http://127.0.97.1:34513/ using document root <none> and password file <none>
I20260929 11:21:13.345752 6619 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:13.345892 6619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:13.346189 6619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:13.349110 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "b67946dde31c42c6bc1f9abc3aa03034"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "f54d164f5d6aea48d6840a7aeb97d6a1"
server_key_iv: "25f013b59789280398321afedf9a852e"
server_key_version: "encryptionkey@0"
I20260929 11:21:13.349928 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b67946dde31c42c6bc1f9abc3aa03034"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "f54d164f5d6aea48d6840a7aeb97d6a1"
server_key_iv: "25f013b59789280398321afedf9a852e"
server_key_version: "encryptionkey@0"
I20260929 11:21:13.354765 6619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:21:13.357947 6634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:13.359130 6619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:21:13.359292 6619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "b67946dde31c42c6bc1f9abc3aa03034"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "f54d164f5d6aea48d6840a7aeb97d6a1"
server_key_iv: "25f013b59789280398321afedf9a852e"
server_key_version: "encryptionkey@0"
I20260929 11:21:13.359483 6619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:13.373651 6619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:13.377576 6619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:13.377861 6619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:13.378767 6619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:13.380050 6619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:13.380154 6619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:13.380306 6619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:13.380378 6619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:13.412739 6619 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35101
I20260929 11:21:13.412761 6746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35101 every 8 connection(s)
I20260929 11:21:13.414011 6619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:13.418627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6619
I20260929 11:21:13.418886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:13.419659 388 external_mini_cluster.cc:1428] Setting key df673c657740c062fcae2050c1bdfc8b
Sep 29 11:21:13 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:13.436200 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:13.416454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47813 (local address 127.0.97.62:33079)
0929 11:21:13.416890 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:13.416895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:13.418056 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 11:21:13.420555 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:13.420560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:13.420655 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:13.420772 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:13.422330 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.422988 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.424484 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.424966 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.425110 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:13.428983 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:13.428991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:13.428999 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:13.429061 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:13.432685 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:13.433704 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:13.433712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:13.433717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:13.433875 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:13.434262 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:13.434269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:13.434276 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:13.434648 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:13.434783 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:13.435434 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 11:21:13.435688 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":245,"threads_started":1}
I20260929 11:21:13.436879 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:13.417066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:47813)
0929 11:21:13.417870 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:13.417935 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:13.420322 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:13.421026 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:13.421034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:13.421605 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:13.422176 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:13.422192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.423138 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.423142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:13.424345 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:13.424356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.425214 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.425222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:13.425439 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:13.426096 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:13.426118 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:13.428783 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:13.432856 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:13.432865 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:13.432882 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:13.433574 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:13.433981 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:13.433984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:13.433986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:13.434157 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:13.434765 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:13.434771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:13.435076 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 11:21:13.435317 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 11:21:13.435679 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":272,"threads_started":1}
I20260929 11:21:13.439146 6747 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260929 11:21:13.439450 6747 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:13.440145 6747 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:21:13.442126 6558 ts_manager.cc:194] Registered new tserver with Master: b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101)
I20260929 11:21:13.443651 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 11:21:13.468400 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:13.781878 6753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:13.782191 6753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:13.782362 6753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:13.789644 6753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:13.789790 6753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:13.789968 6753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:13.799664 6753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:13.801105 6753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:13.802469 6753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:13.815402 6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:13.815438 6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:13.815459 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:13.816154 6753 server_base.cc:1034] running on GCE node
I20260929 11:21:13.816762 6753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:13.817797 6753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:13.819006 6753 hybrid_clock.cc:648] HybridClock initialized: now 1790680873818967 us; error 33 us; skew 500 ppm
I20260929 11:21:13.819267 6753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:13 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:13.822314 6753 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 11:21:13.823585 6753 webserver.cc:460] Webserver started at http://127.0.97.2:45705/ using document root <none> and password file <none>
I20260929 11:21:13.824188 6753 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:13.824285 6753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:13.824517 6753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:13.827293 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "8cf6600aabbe4cfea93c14e272f75a14"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "6670d3de481b43a85b0b3b99ef9edf59"
server_key_iv: "c3717fe5d4e8b71d6d3d332b22f61ae1"
server_key_version: "encryptionkey@0"
I20260929 11:21:13.828003 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8cf6600aabbe4cfea93c14e272f75a14"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "6670d3de481b43a85b0b3b99ef9edf59"
server_key_iv: "c3717fe5d4e8b71d6d3d332b22f61ae1"
server_key_version: "encryptionkey@0"
I20260929 11:21:13.833107 6753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:21:13.836498 6768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:13.838042 6753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 11:21:13.838186 6753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "8cf6600aabbe4cfea93c14e272f75a14"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "6670d3de481b43a85b0b3b99ef9edf59"
server_key_iv: "c3717fe5d4e8b71d6d3d332b22f61ae1"
server_key_version: "encryptionkey@0"
I20260929 11:21:13.838336 6753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:13.853832 6753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:13.857559 6753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:13.857785 6753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:13.858707 6753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:13.860132 6753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:13.860224 6753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:13.860373 6753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:13.860442 6753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:13.894145 6753 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40713
I20260929 11:21:13.894176 6880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40713 every 8 connection(s)
I20260929 11:21:13.895413 6753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:13.897286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6753
I20260929 11:21:13.897497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:13.898031 388 external_mini_cluster.cc:1428] Setting key 4c5af9f462316982712111b3c5b4f573
Sep 29 11:21:13 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:13.918072 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:13.898066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42973 (local address 127.0.97.62:33079)
0929 11:21:13.898279 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:13.898285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:13.899463 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 11:21:13.902135 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:13.902140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:13.902200 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:13.902283 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:13.904289 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.905119 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.907443 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.908151 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.908292 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:13.912301 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:13.912306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:13.912310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:13.912341 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:13.914843 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:13.915740 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:13.915744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:13.915746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:13.915874 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:13.916316 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:13.916321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:13.916325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:13.916580 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:13.916796 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:13.917480 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 11:21:13.917730 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 11:21:13.919072 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:13.898611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:42973)
0929 11:21:13.899323 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:13.899360 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:13.901851 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:13.902539 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:13.902553 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:13.903308 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:13.904124 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:13.904148 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.905287 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.905295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:13.907282 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:13.907292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:13.908343 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:13.908348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:13.908708 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:13.909373 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:13.909414 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:13.912106 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:13.914972 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:13.914982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:13.915002 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:13.915618 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:13.915967 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:13.915970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:13.915972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:13.916188 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:13.916758 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:13.916766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:13.917099 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 11:21:13.917416 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 11:21:13.917937 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:21:13.920463 6881 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260929 11:21:13.920831 6881 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:13.921604 6881 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260929 11:21:13.923069 6558 ts_manager.cc:194] Registered new tserver with Master: 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713)
I20260929 11:21:13.923858 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 11:21:13.947325 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:14.259485 6886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:14.259744 6886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:14.259830 6886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:14.267201 6886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:14.267318 6886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:14.267427 6886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:14.276793 6886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:14.277961 6886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:14.279117 6886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:14.291770 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:14.291816 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:14.291862 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:14.292574 6886 server_base.cc:1034] running on GCE node
I20260929 11:21:14.293213 6886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:14.294116 6886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:14.295310 6886 hybrid_clock.cc:648] HybridClock initialized: now 1790680874295267 us; error 31 us; skew 500 ppm
I20260929 11:21:14.295571 6886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:14.298653 6886 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 11:21:14.299968 6886 webserver.cc:460] Webserver started at http://127.0.97.3:35835/ using document root <none> and password file <none>
I20260929 11:21:14.300710 6886 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:14.300839 6886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:14.301199 6886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:14.304191 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "4855b9133cb94dc8970dad7192a7e497"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "cd3685a22e9b8ca36ea7f4c2f4730920"
server_key_iv: "68819b7ac0a9c31189828e5363ab330d"
server_key_version: "encryptionkey@0"
I20260929 11:21:14.304967 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4855b9133cb94dc8970dad7192a7e497"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "cd3685a22e9b8ca36ea7f4c2f4730920"
server_key_iv: "68819b7ac0a9c31189828e5363ab330d"
server_key_version: "encryptionkey@0"
I20260929 11:21:14.310149 6886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:21:14.313496 6901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:14.314841 6886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:21:14.315008 6886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "4855b9133cb94dc8970dad7192a7e497"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "cd3685a22e9b8ca36ea7f4c2f4730920"
server_key_iv: "68819b7ac0a9c31189828e5363ab330d"
server_key_version: "encryptionkey@0"
I20260929 11:21:14.315196 6886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:14.331144 6886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:14.334991 6886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:14.335280 6886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:14.336207 6886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:14.337594 6886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:14.337703 6886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:14.337850 6886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:14.337921 6886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:14.370077 6886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37223
I20260929 11:21:14.370115 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37223 every 8 connection(s)
I20260929 11:21:14.371363 6886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:14.375296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 6886
I20260929 11:21:14.375479 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:14.376053 388 external_mini_cluster.cc:1428] Setting key e71caf8804b1a689448ddee8de59230a
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:14.390525 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.373885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47351 (local address 127.0.97.62:33079)
0929 11:21:14.374091 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.374097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.375338 (+ 1241us) server_negotiation.cc:407] Connection header received
0929 11:21:14.377761 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.377765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.377864 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.377955 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.379524 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.380213 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.381702 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.382073 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.382167 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.385208 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.385213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.385216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.385244 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.387728 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.388326 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.388330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.388332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.388432 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.388864 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.388867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.388869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.389109 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.389221 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.390035 (+ 814us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.390208 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:21:14.391553 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.374402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:47351)
0929 11:21:14.375153 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:14.375204 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:14.377452 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:14.378206 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:14.378214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:14.378763 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:14.379389 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.379405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.380332 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.380336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.381577 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.381587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.382247 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.382251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.382427 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.382993 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:14.383012 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:14.385031 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:14.387845 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.387850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.387861 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.388214 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.388516 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.388519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.388521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.388767 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.389223 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:14.389231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:14.389612 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 11:21:14.389905 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 11:21:14.390394 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":269,"threads_started":1}
I20260929 11:21:14.392940 7014 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260929 11:21:14.393218 7014 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:14.393889 7014 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260929 11:21:14.395221 6558 ts_manager.cc:194] Registered new tserver with Master: 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:14.396003 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47351
I20260929 11:21:14.403002 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260929 11:21:14.447410 6747 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 11:21:14.452311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35157
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33079,127.0.97.61:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:14.766736 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:14.767076 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:14.767261 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:14.774248 7019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:14.774394 7019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:14.774502 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:14.774606 7019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:14.774695 7019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:14.784153 7019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33079,127.0.97.61:35157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
W20260929 11:21:14.784560 7019 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:33079,127.0.97.61:35157'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 11:21:14.785624 7019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:14.786933 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:14.798221 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:14.798314 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:14.798242 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:14.798933 7019 server_base.cc:1034] running on GCE node
I20260929 11:21:14.799742 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:14.801164 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:14.802393 7019 hybrid_clock.cc:648] HybridClock initialized: now 1790680874802338 us; error 37 us; skew 500 ppm
I20260929 11:21:14.802767 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:14.805634 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 11:21:14.806982 7019 webserver.cc:460] Webserver started at http://127.0.97.61:35113/ using document root <none> and password file <none>
I20260929 11:21:14.807648 7019 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:14.807771 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:14.808075 7019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:14.811318 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/instance:
uuid: "08047313636d4bdf8e6ede057fa5c893"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "35cc05687ad1d9e4960fac23118a435e"
server_key_iv: "a933682b857aa9363c44611f5ee5ef1e"
server_key_version: "encryptionkey@0"
I20260929 11:21:14.812110 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal/instance:
uuid: "08047313636d4bdf8e6ede057fa5c893"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "35cc05687ad1d9e4960fac23118a435e"
server_key_iv: "a933682b857aa9363c44611f5ee5ef1e"
server_key_version: "encryptionkey@0"
I20260929 11:21:14.816974 7019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 11:21:14.820538 7034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:14.821902 7019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:14.822077 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
uuid: "08047313636d4bdf8e6ede057fa5c893"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "35cc05687ad1d9e4960fac23118a435e"
server_key_iv: "a933682b857aa9363c44611f5ee5ef1e"
server_key_version: "encryptionkey@0"
I20260929 11:21:14.822281 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:14.839162 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:14.843122 7019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:14.843382 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:14.861060 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35157
I20260929 11:21:14.861078 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35157 every 8 connection(s)
I20260929 11:21:14.862618 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
I20260929 11:21:14.866561 7086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:14.871345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 7019
I20260929 11:21:14.873299 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 11:21:14.890559 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.875841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.0.97.62:33079)
0929 11:21:14.875999 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.876003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.876635 (+ 632us) server_negotiation.cc:407] Connection header received
0929 11:21:14.877046 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.877051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.877129 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.877241 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.878368 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.879065 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.880459 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.880928 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.881056 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.884587 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.884625 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.884630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.884669 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.887405 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.888231 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.888235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.888237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.888349 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.888823 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.888826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.888828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.889256 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.889398 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.890125 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.890274 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:21:14.891779 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.875782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:36576)
0929 11:21:14.876482 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:14.876515 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:14.876834 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:14.877313 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:14.877320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:14.877884 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:14.878211 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.878226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.879188 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.879193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.880321 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.880332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.881145 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.881151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.881394 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.882062 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:14.882090 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:14.884396 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:14.887527 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.887535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.887547 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.888119 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.888450 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.888457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.888461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.888715 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.889378 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:14.889384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:14.889761 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 11:21:14.890046 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 11:21:14.890511 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":16,"thread_start_us":373,"threads_started":1}
I20260929 11:21:14.891799 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.873599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.97.61:35157)
0929 11:21:14.874404 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.874414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.874453 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:21:14.876512 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.876535 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.876950 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.877153 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.878518 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.879644 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.880810 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.881364 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.881505 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.885563 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.885576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.885592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.885629 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.888825 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.889511 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.889518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.889521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.889657 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.889989 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.889996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.890000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.890362 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.890637 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.891016 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.891213 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":629,"spinlock_wait_cycles":22912,"thread_start_us":373,"threads_started":1}
I20260929 11:21:14.894541 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 11:21:14.905915 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.895140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57490)
0929 11:21:14.895314 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:14.895340 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:14.895446 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:14.895773 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:14.895775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:14.896046 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:14.896160 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.896167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.897436 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.897439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.898230 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.898240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.899140 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.899143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.899284 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.899727 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:14.899742 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:14.901600 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:14.904081 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.904085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.904087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.904408 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.904754 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.904757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.904759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.904868 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.905273 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:14.905276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:14.905370 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 11:21:14.905480 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:21:14.905658 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 11:21:14.906054 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.895342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.0.97.61:35157)
0929 11:21:14.895502 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.895525 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.895543 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:21:14.895589 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.895594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.895655 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.895749 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.896283 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.897326 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.898355 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.899009 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.899122 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.901778 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.901784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.901788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.901833 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.903980 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.904498 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.904502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.904504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.904655 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.904951 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.904954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.904956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.905171 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.905267 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.905558 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.905747 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":65}
I20260929 11:21:14.910444 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Bootstrap starting.
I20260929 11:21:14.913465 7086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:14.914436 7086 log.cc:824] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:14.916909 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: No bootstrap required, opened a new log
I20260929 11:21:14.923714 7086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } }
I20260929 11:21:14.924043 7086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:14.924139 7086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08047313636d4bdf8e6ede057fa5c893, State: Initialized, Role: FOLLOWER
I20260929 11:21:14.924803 7086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } }
I20260929 11:21:14.925624 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } }
I20260929 11:21:14.925943 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:14.926666 7086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:14.926950 6881 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
W20260929 11:21:14.936004 7103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:21:14.936156 7103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:21:14.937836 7019 master_runner.cc:428] Detected that this master 08047313636d4bdf8e6ede057fa5c893 is joining an existing cluster
I20260929 11:21:14.937939 7019 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:35157
I20260929 11:21:14.958487 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.941803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.0.97.62:33079)
0929 11:21:14.941991 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.941998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.942601 (+ 603us) server_negotiation.cc:407] Connection header received
0929 11:21:14.942898 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.942906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.942991 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.943055 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.944516 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.945702 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.947877 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.948492 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.948575 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.951513 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.951518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.951523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.951555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.955280 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.956094 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.956100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.956104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.956251 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.956754 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.956759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.956762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.957114 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.957258 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.957887 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.958116 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:21:14.958613 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.942656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.0.97.61:35157)
0929 11:21:14.942908 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.942921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.943520 (+ 599us) server_negotiation.cc:407] Connection header received
0929 11:21:14.943696 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.943701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.943759 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.943851 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.944969 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.945974 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.947854 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.948488 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.948575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.951513 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.951518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.951523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.951555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.955425 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.956263 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.956270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.956276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.956410 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.956847 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.956854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.956860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.957277 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.957406 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.957912 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.958144 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:21:14.959743 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.942615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57492)
0929 11:21:14.943427 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:14.943440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:14.943587 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:14.943874 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:14.943878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:14.944472 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:14.944815 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.944827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.946231 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.946237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.947734 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.947741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.948917 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.948930 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.949149 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.950017 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:14.950035 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:14.951334 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:14.955552 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.955559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.955570 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.956151 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.956535 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.956542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.956548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.956744 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.957491 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:14.957497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:14.957681 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 11:21:14.958818 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0929 11:21:14.959132 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":60,"spinlock_wait_cycles":2432,"thread_start_us":553,"threads_started":1}
I20260929 11:21:14.959769 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.941837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:36586)
0929 11:21:14.942499 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:14.942523 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:14.942701 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:14.943264 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:14.943269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:14.943945 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:14.944368 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.944381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.945868 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.945873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.947719 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.947732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.948943 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.948949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.949182 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.949950 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:14.949982 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:14.951334 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:14.955430 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.955436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.955439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.955937 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.956398 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.956405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.956413 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.956575 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.957351 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:14.957356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:14.957559 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 11:21:14.958908 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0929 11:21:14.959325 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":209,"spinlock_wait_cycles":4864,"thread_start_us":339,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 11:21:14 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 11:21:14.986804 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.967027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57498)
0929 11:21:14.970087 (+ 3060us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:14.970122 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:14.970306 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:14.970764 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:14.970771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:14.971428 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:14.971622 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.971631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.973006 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.973013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.974554 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:14.974571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.975633 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.975641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:14.975846 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.976605 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:14.976631 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:14.979342 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:14.983616 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.983630 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.983638 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.984192 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.984665 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:14.984670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:14.984673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:14.984877 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:14.985556 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:14.985565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:14.985755 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 11:21:14.985961 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 11:21:14.986315 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2882,"thread_start_us":2758,"threads_started":1}
I20260929 11:21:14.986864 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:14.967202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.97.61:35157)
0929 11:21:14.967411 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:14.967417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:14.970215 (+ 2798us) server_negotiation.cc:407] Connection header received
0929 11:21:14.970516 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:14.970522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:14.970615 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:14.970731 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:14.971759 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.972866 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.974709 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:14.975468 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:14.975602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:14.979537 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:14.979546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:14.979554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:14.979595 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:14.983477 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.984337 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.984345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.984352 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.984507 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:14.984995 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:14.985003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:14.985009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:14.985409 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:14.985558 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:14.986140 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 11:21:14.986399 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:21:14.989571 6558 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:35157: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08047313636d4bdf8e6ede057fa5c893"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 35157
}
attrs {
promote: true
}
}
dest_uuid: "08ec214716054f739cd17e77b4e5473d"
cas_config_opid_index: -1
I20260929 11:21:14.990756 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } }
I20260929 11:21:14.994915 7061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:14.997782 7061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Refusing update from remote peer 08ec214716054f739cd17e77b4e5473d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260929 11:21:14.999008 7116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:14.998504 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.000409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46339
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:21:15.017560 7115 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:35157
I20260929 11:21:15.018344 7019 master.cc:537] Master@127.0.97.61:35157 shutting down...
I20260929 11:21:15.019671 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.019992 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:15.037510 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.060410 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.061005 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:21:15.060871 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.061268 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:21:15.061544 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec214716054f739cd17e77b4e5473d. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.061786 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:21:15.064626 7019 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 LEARNER]: Raft consensus shutting down.
I20260929 11:21:15.065096 7019 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 LEARNER]: Raft consensus is shut down!
I20260929 11:21:15.065255 7019 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: stopping tablet replica
I20260929 11:21:15.082723 7019 master.cc:552] Master@127.0.97.61:35157 shutdown complete.
I20260929 11:21:15.082823 7019 master_runner.cc:305] Clearing existing system tablet
I20260929 11:21:15.083284 7019 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:15.083397 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:15.086575 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 11:21:15.088341 7128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:15.088693 7019 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:15.088977 7019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 11:21:15.089064 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
uuid: "08047313636d4bdf8e6ede057fa5c893"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "35cc05687ad1d9e4960fac23118a435e"
server_key_iv: "a933682b857aa9363c44611f5ee5ef1e"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.093925 7019 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:15.100463 7019 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 11:21:15.100682 7019 log.cc:1196] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 11:21:15.101222 7019 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Deleting consensus metadata
I20260929 11:21:15.101485 7019 master_runner.cc:315] Copying system tablet from 127.0.97.62:33079
I20260929 11:21:15.109848 7019 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33079
I20260929 11:21:15.127800 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.110680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.0.97.62:33079)
0929 11:21:15.110899 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.110906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.113234 (+ 2328us) server_negotiation.cc:407] Connection header received
0929 11:21:15.113417 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.113423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.113527 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.113636 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.114813 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.115914 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.117893 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.118637 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.118762 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.121352 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.121359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.121364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.121403 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.124919 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.125685 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.125690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.125693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.125844 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.126266 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.126270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.126273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.126651 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.126788 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.127243 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.127481 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:21:15.127998 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.110626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:36598)
0929 11:21:15.113098 (+ 2472us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.113121 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.113283 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.113702 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.113707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.114319 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.114647 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.114659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.116077 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.116082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.117744 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.117758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.118820 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.118828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.119111 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.119907 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.119938 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.121150 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.125048 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.125053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.125057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.125548 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.125957 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.125962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.125966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.126142 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.126767 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.126772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.126961 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.127109 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.127520 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2337,"thread_start_us":2146,"threads_started":1}
I20260929 11:21:15.128903 6578 tablet_copy_service.cc:140] P 08ec214716054f739cd17e77b4e5473d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08047313636d4bdf8e6ede057fa5c893 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36598)
I20260929 11:21:15.129137 6578 tablet_copy_service.cc:161] P 08ec214716054f739cd17e77b4e5473d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08047313636d4bdf8e6ede057fa5c893 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36598: session id = 08047313636d4bdf8e6ede057fa5c893-00000000000000000000000000000000
I20260929 11:21:15.133612 6578 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 11:21:15.135663 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 11:21:15.142421 7019 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: tablet copy: Starting download of 0 data blocks...
I20260929 11:21:15.142704 7019 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: tablet copy: Starting download of 1 WAL segments...
I20260929 11:21:15.145342 7019 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 11:21:15.149654 6578 tablet_copy_service.cc:342] P 08ec214716054f739cd17e77b4e5473d: Request end of tablet copy session 08047313636d4bdf8e6ede057fa5c893-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36598
I20260929 11:21:15.149899 6578 tablet_copy_service.cc:433] P 08ec214716054f739cd17e77b4e5473d: ending tablet copy session 08047313636d4bdf8e6ede057fa5c893-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08047313636d4bdf8e6ede057fa5c893
I20260929 11:21:15.160691 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:15.163352 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:15.165973 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:15.182294 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:15.182363 7019 server_base.cc:1034] running on GCE node
I20260929 11:21:15.182945 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:15.183619 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:15.184823 7019 hybrid_clock.cc:648] HybridClock initialized: now 1790680875184713 us; error 85 us; skew 500 ppm
I20260929 11:21:15.185048 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:15.187219 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 11:21:15.189951 7019 webserver.cc:460] Webserver started at http://127.0.97.61:39983/ using document root <none> and password file <none>
I20260929 11:21:15.190335 7019 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:15.190423 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:15.193810 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 11:21:15.195729 7145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:15.196148 7019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 11:21:15.196265 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
uuid: "08047313636d4bdf8e6ede057fa5c893"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "35cc05687ad1d9e4960fac23118a435e"
server_key_iv: "a933682b857aa9363c44611f5ee5ef1e"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.196431 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:15.217041 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:15.221922 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:15.255842 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35157
I20260929 11:21:15.255915 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35157 every 8 connection(s)
I20260929 11:21:15.256865 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
I20260929 11:21:15.262156 7197 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:21:15.265944 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Bootstrap starting.
I20260929 11:21:15.280283 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:21:15.280786 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: Bootstrap complete.
I20260929 11:21:15.281786 7197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } }
I20260929 11:21:15.282027 7197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08047313636d4bdf8e6ede057fa5c893, State: Initialized, Role: LEARNER
I20260929 11:21:15.282284 7197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } }
I20260929 11:21:15.289016 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.289325 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:21:15.289952 7197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:15.309113 7211 catalog_manager.cc:1259] Loaded cluster ID: 4d275977890e45628abadcad783f75db
I20260929 11:21:15.309224 7211 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: loading cluster ID for follower catalog manager: success
I20260929 11:21:15.313246 7211 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: acquiring CA information for follower catalog manager: success
I20260929 11:21:15.315742 7211 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:21:15.398326 7014 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
W20260929 11:21:15.403774 7118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:15.404030 7118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:15.404189 7118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:15.411402 7118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:15.411494 7118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:15.411610 7118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:15.411706 7118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:15.411780 7118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:15.421615 7118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:15.422874 7118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:15.424116 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:15.435544 7217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:15.435629 7219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:15.436121 7216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:15.436316 7118 server_base.cc:1034] running on GCE node
I20260929 11:21:15.437215 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:15.439078 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:15.440336 7118 hybrid_clock.cc:648] HybridClock initialized: now 1790680875440241 us; error 96 us; skew 500 ppm
I20260929 11:21:15.440760 7118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:15.443817 7118 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 11:21:15.445235 7118 webserver.cc:460] Webserver started at http://127.0.97.60:33571/ using document root <none> and password file <none>
I20260929 11:21:15.445851 7118 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:15.445942 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:15.446197 7118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:15.448974 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/instance:
uuid: "c52b984fa50548ef81ac294b248bdc16"
format_stamp: "Formatted at 2026-09-29 11:21:15 on dist-test-slave-srst"
server_key: "39902bc2c24fa467def6790d8d160be8"
server_key_iv: "b390c13a272bf3924bddadef786ad5bc"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.449697 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal/instance:
uuid: "c52b984fa50548ef81ac294b248bdc16"
format_stamp: "Formatted at 2026-09-29 11:21:15 on dist-test-slave-srst"
server_key: "39902bc2c24fa467def6790d8d160be8"
server_key_iv: "b390c13a272bf3924bddadef786ad5bc"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.455103 7118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:21:15.458385 7226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:15.459632 7118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:21:15.459800 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
uuid: "c52b984fa50548ef81ac294b248bdc16"
format_stamp: "Formatted at 2026-09-29 11:21:15 on dist-test-slave-srst"
server_key: "39902bc2c24fa467def6790d8d160be8"
server_key_iv: "b390c13a272bf3924bddadef786ad5bc"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.459977 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:15.482647 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:15.486690 7118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:15.486981 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:15.506223 7118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46339
I20260929 11:21:15.506258 7277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46339 every 8 connection(s)
I20260929 11:21:15.507483 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
I20260929 11:21:15.509688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 7118
I20260929 11:21:15.511731 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:15.519263 7278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:15.536259 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.511893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.97.60:46339)
0929 11:21:15.512679 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.512690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.512728 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:21:15.514533 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.514554 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.514925 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.515311 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.516804 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.518473 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.519774 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.520469 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.520670 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.528463 (+ 7793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.528472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.528485 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.528528 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.532120 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.532921 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.532929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.532937 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.533106 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.533454 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.533460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.533463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.534058 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.534349 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.534777 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.535274 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":344,"threads_started":1}
I20260929 11:21:15.536384 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.521808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.0.97.62:33079)
0929 11:21:15.522039 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.522044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.522405 (+ 361us) server_negotiation.cc:407] Connection header received
0929 11:21:15.522534 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.522540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.522612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.522686 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.523610 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.524637 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.526162 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.526850 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.526948 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.530494 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.530502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.530510 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.530550 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.533705 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.534349 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.534354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.534357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.534466 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.534869 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.534876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.534882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.535204 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.535289 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.535814 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.536023 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:21:15.536491 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.521713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:36602)
0929 11:21:15.522265 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.522309 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.522434 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.522743 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.522747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.523276 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.523496 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.523503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.524763 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.524767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.525725 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.525736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.527061 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.527068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.527267 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.527896 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.527917 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.530312 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.533834 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.533842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.533847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.534220 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.534577 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.534582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.534587 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.534712 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.535372 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.535377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.535516 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.535703 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.536013 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260929 11:21:15.538580 7278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 11:21:15.549154 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.549428 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:15.553018 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.539161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57502)
0929 11:21:15.539315 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.539331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.539414 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.540489 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.540492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.540839 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.540956 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.540963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.542238 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.542242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.543180 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.543190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.544180 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.544184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.544313 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.544815 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.544831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.546532 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.549492 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.549496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.549499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.550189 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.550748 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.550756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.550762 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.550942 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.551761 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.551769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.551950 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.552145 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.552523 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:21:15.554389 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.539347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.97.61:35157)
0929 11:21:15.539812 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.539816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.539830 (+ 14us) server_negotiation.cc:407] Connection header received
0929 11:21:15.539876 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.539879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.539932 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.540468 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.541076 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.542122 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.543300 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.544072 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.544164 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.546701 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.546706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.546709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.546737 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.549334 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.550383 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.550391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.550397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.550600 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.551086 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.551094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.551100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.551504 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.551693 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.553518 (+ 1825us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.553797 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":262,"threads_started":1}
I20260929 11:21:15.556075 7278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:21:15.561730 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.549810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.97.61:35157)
0929 11:21:15.555029 (+ 5219us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.555037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.556912 (+ 1875us) server_negotiation.cc:407] Connection header received
0929 11:21:15.557106 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.557111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.557198 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.557310 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:15.557723 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.558425 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.559969 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.560848 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.560982 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.561192 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.561365 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.561479 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5057,"thread_start_us":5701,"threads_started":1}
I20260929 11:21:15.561755 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.549318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57514)
0929 11:21:15.556782 (+ 7464us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.556819 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.556952 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.557363 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.557368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.557393 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:15.557569 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.557577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.558556 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.558562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.559787 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.559798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.560995 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.561000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.561161 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.561225 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.561347 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.561473 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7328,"thread_start_us":7157,"threads_started":1}
I20260929 11:21:15.562575 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec214716054f739cd17e77b4e5473d. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.562809 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: LEARNER
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 11:21:15.563411 7284 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: attempting to promote NON_VOTER 08047313636d4bdf8e6ede057fa5c893 to VOTER
I20260929 11:21:15.564236 7284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } }
I20260929 11:21:15.566073 7172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 LEARNER]: Refusing update from remote peer 08ec214716054f739cd17e77b4e5473d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 11:21:15.566601 7285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 11:21:15.569283 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.556918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.0.1:53832)
0929 11:21:15.557102 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.557123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.557232 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.557503 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.557506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.557829 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.557959 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.557967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.558844 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.558852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.560010 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.560018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.560638 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.560643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.560838 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.561331 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.561347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.563583 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.566914 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.566918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.566922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.567356 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.567689 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.567693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.567696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.567912 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.568362 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.568366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.568486 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.568631 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.568876 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 11:21:15.569341 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.556909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.97.60:46339)
0929 11:21:15.557060 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.557066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.557183 (+ 117us) server_negotiation.cc:407] Connection header received
0929 11:21:15.557335 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.557339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.557404 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.557491 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.558109 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.558735 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.560128 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.560496 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.560637 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.563759 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.563763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.563766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.563789 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.566761 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.567470 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.567474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.567476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.567584 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.568013 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.568016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.568018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.568255 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.568407 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.568695 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.569005 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":41}
I20260929 11:21:15.569638 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } }
I20260929 11:21:15.570102 7172 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } }
I20260929 11:21:15.573012 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } }
I20260929 11:21:15.573206 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:15.573185 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } }
I20260929 11:21:15.573206 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } }
I20260929 11:21:15.573406 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:15.573457 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:15.575275 7278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Bootstrap starting.
I20260929 11:21:15.578271 7278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:15.579190 7278 log.cc:824] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:15.581605 7278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: No bootstrap required, opened a new log
I20260929 11:21:15.588464 7278 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } }
I20260929 11:21:15.588802 7278 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:15.588891 7278 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52b984fa50548ef81ac294b248bdc16, State: Initialized, Role: FOLLOWER
I20260929 11:21:15.589447 7278 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } }
I20260929 11:21:15.590581 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } } }
I20260929 11:21:15.590864 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:15.591650 7278 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:21:15.600824 7302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:21:15.600970 7302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:21:15.604715 7118 master_runner.cc:428] Detected that this master c52b984fa50548ef81ac294b248bdc16 is joining an existing cluster
I20260929 11:21:15.604838 7118 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:46339
I20260929 11:21:15.626576 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.608935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.0.97.60:46339)
0929 11:21:15.609242 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.609248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.609950 (+ 702us) server_negotiation.cc:407] Connection header received
0929 11:21:15.610127 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.610133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.610205 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.610327 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.611819 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.613031 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.615835 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.616524 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.616682 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.619477 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.619483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.619487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.619521 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.623199 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.623994 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.623999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.624002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.624418 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.624862 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.624867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.624877 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.625201 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.625362 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.625947 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.626160 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 11:21:15.627797 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.608982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.0.1:53846)
0929 11:21:15.609830 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.609854 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.610010 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.610360 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.610371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.611198 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.611652 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.611667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.613221 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.613241 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.615669 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.615688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.616929 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.616936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.617182 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.617930 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.617959 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.619268 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.623363 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.623368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.623372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.623872 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.624561 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.624566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.624570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.624742 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.625452 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.625456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.625692 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.627077 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.627344 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":542,"threads_started":1}
I20260929 11:21:15.632965 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.609434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:36610)
0929 11:21:15.616102 (+ 6668us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.616115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.616259 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.616785 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.616790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.617584 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.618070 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.618087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.619946 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.619954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.622180 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.622197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.623681 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.623687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.624316 (+ 629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.624948 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.624974 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.626045 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.629776 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.629781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.629785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.630211 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.630641 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.630644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.630647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.630842 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.631469 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.631474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.631683 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.631883 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.632234 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6575,"thread_start_us":6931,"threads_started":1}
I20260929 11:21:15.633309 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.608858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.97.62:33079)
0929 11:21:15.609067 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.609073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.616214 (+ 7141us) server_negotiation.cc:407] Connection header received
0929 11:21:15.616400 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.616406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.616480 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.616575 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.618270 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.619666 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.622374 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.623264 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.623373 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.626294 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.626300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.626304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.626348 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.629566 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.630327 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.630332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.630335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.630510 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.630989 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.630993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.630996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.631319 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.631490 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.632068 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.632280 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:21:15.638890 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.616959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.97.61:35157)
0929 11:21:15.617123 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.617129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.624523 (+ 7394us) server_negotiation.cc:407] Connection header received
0929 11:21:15.624800 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.624811 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.624912 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.624988 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.626305 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.627415 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.629535 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.630133 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.630222 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.632499 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.632912 (+ 413us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.632919 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.632962 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.636052 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.636753 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.636758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.636762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.636907 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.637316 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.637334 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.637337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.637659 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.637784 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.638347 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.638550 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 11:21:15.639748 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.617159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57530)
0929 11:21:15.624421 (+ 7262us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.624442 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.624643 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.625194 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.625199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.625800 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.626133 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.626145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.627590 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.627595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.629368 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.629382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.630499 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.630504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.630668 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.631244 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.631265 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.632282 (+ 1017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.636188 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.636212 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.636221 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.636635 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.637028 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.637032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.637035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.637204 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.637838 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.637843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.638056 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.639054 (+ 998us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.639324 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6518,"thread_start_us":6667,"threads_started":1}
I20260929 11:21:15.641848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6619
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 11:21:15.648924 7291 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:15.649245 7291 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } }
I20260929 11:21:15.651083 7291 leader_election.cc:288] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079), 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61:35157)
I20260929 11:21:15.651870 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.637975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.0.1:53860)
0929 11:21:15.638236 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.638253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.638349 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.638697 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.638704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.639168 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.639297 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.639304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.640547 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.640554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.641837 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.641852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.642831 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.642839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.643016 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.643591 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.643620 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.645963 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.649219 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.649225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.649230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.649716 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.650118 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.650122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.650125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.650300 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.650935 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.650942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.651098 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.651242 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.651508 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 11:21:15.652205 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52b984fa50548ef81ac294b248bdc16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ec214716054f739cd17e77b4e5473d" is_pre_election: true
I20260929 11:21:15.652747 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.638103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.0.97.60:46339)
0929 11:21:15.638236 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.638243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.638297 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:21:15.638497 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.638502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.638578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.638683 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.639427 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.640385 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.641985 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.642668 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.642797 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.646205 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.646210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.646215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.646250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.649061 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.649845 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.649849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.649852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.649999 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.650414 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.650418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.650421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.650734 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.650863 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.652215 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.652433 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:21:15.653029 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52b984fa50548ef81ac294b248bdc16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08047313636d4bdf8e6ede057fa5c893" is_pre_election: true
I20260929 11:21:15.653882 7230 leader_election.cc:302] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c52b984fa50548ef81ac294b248bdc16; no voters: 08047313636d4bdf8e6ede057fa5c893, 08ec214716054f739cd17e77b4e5473d
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 11:21:15.655095 7291 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 11:21:15.655189 6558 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:46339: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c52b984fa50548ef81ac294b248bdc16"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 46339
}
attrs {
promote: true
}
}
dest_uuid: "08ec214716054f739cd17e77b4e5473d"
cas_config_opid_index: 6
I20260929 11:21:15.655886 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } }
I20260929 11:21:15.658427 7172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Refusing update from remote peer 08ec214716054f739cd17e77b4e5473d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 11:21:15.658700 7253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:15.662465 7253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Refusing update from remote peer 08ec214716054f739cd17e77b4e5473d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 11:21:15.662633 7116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 11:21:15.663654 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.642956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.0.97.1:35101)
0929 11:21:15.643965 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.643975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.644005 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:21:15.644070 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.644078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.644371 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.644539 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.646093 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.647625 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.649515 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.650392 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.654609 (+ 4217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.656114 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.656124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.656140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.656170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.659837 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.660651 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.660660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.660663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.660798 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.661245 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.661252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.661256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.661626 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.662779 (+ 1153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.662959 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.663248 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":680,"threads_started":1}
I20260929 11:21:15.663913 7116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:15.665678 6742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:15.666168 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c52b984fa50548ef81ac294b248bdc16 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.669119 7172 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c52b984fa50548ef81ac294b248bdc16 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.669584 7285 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:46339
I20260929 11:21:15.670143 7118 master.cc:537] Master@127.0.97.60:46339 shutting down...
I20260929 11:21:15.672446 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.673175 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:15.672529 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.673609 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:15.676376 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.676750 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:15.690254 7252 raft_consensus.cc:858] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } }
W20260929 11:21:15.690586 7252 raft_consensus.cc:1565] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } }
I20260929 11:21:15.706612 7252 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.720969 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.721441 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 11:21:15.721817 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.722185 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 11:21:15.721824 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec214716054f739cd17e77b4e5473d. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: true } } }
I20260929 11:21:15.722571 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 11:21:15.723716 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.723964 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
W20260929 11:21:15.725044 6543 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 11:21:15.725178 7118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 11:21:15.725616 7118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 11:21:15.725842 7118 tablet_replica.cc:330] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: stopping tablet replica
W20260929 11:21:15.729815 6543 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d -> Peer c52b984fa50548ef81ac294b248bdc16 (127.0.97.60:46339): Couldn't send request to peer c52b984fa50548ef81ac294b248bdc16. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:21:15.743675 7118 master.cc:552] Master@127.0.97.60:46339 shutdown complete.
I20260929 11:21:15.743777 7118 master_runner.cc:305] Clearing existing system tablet
I20260929 11:21:15.744119 7118 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:15.744244 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:15.746508 7118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 11:21:15.747895 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:15.748198 7118 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:15.748469 7118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 11:21:15.748566 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
uuid: "c52b984fa50548ef81ac294b248bdc16"
format_stamp: "Formatted at 2026-09-29 11:21:15 on dist-test-slave-srst"
server_key: "39902bc2c24fa467def6790d8d160be8"
server_key_iv: "b390c13a272bf3924bddadef786ad5bc"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.751511 7118 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:15.755896 7118 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 11:21:15.756053 7118 log.cc:1196] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 11:21:15.756498 7118 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Deleting consensus metadata
I20260929 11:21:15.756754 7118 master_runner.cc:315] Copying system tablet from 127.0.97.62:33079
I20260929 11:21:15.759198 7118 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33079
I20260929 11:21:15.771479 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.759914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.0.97.62:33079)
0929 11:21:15.760130 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:15.760139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:15.761110 (+ 971us) server_negotiation.cc:407] Connection header received
0929 11:21:15.761334 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:15.761339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:15.761413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:15.761516 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:15.762276 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.763033 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.764187 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.764581 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.764726 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.766488 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:15.766494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:15.766499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:15.766538 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:15.769031 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.769600 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.769611 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.769615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.769726 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:15.770068 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:15.770090 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:15.770094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:15.770353 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:15.770520 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:15.770873 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 11:21:15.771118 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:21:15.771505 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:15.759889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:36616)
0929 11:21:15.760978 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:15.761000 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:15.761171 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:15.761543 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:15.761547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:15.761935 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:15.762127 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.762135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.763171 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.763177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.764051 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:15.764061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:15.764745 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:15.764751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:15.764919 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:15.765602 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:15.765620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:15.766293 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:15.769160 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.769167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.769171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.769490 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.769824 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:15.769828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:15.769832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:15.769970 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:15.770471 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:15.770475 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:15.770633 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 11:21:15.770769 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:21:15.771085 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":867,"threads_started":1}
I20260929 11:21:15.772354 6578 tablet_copy_service.cc:140] P 08ec214716054f739cd17e77b4e5473d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c52b984fa50548ef81ac294b248bdc16 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36616)
I20260929 11:21:15.772518 6578 tablet_copy_service.cc:161] P 08ec214716054f739cd17e77b4e5473d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c52b984fa50548ef81ac294b248bdc16 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36616: session id = c52b984fa50548ef81ac294b248bdc16-00000000000000000000000000000000
I20260929 11:21:15.775372 6578 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 11:21:15.777002 7118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 11:21:15.781515 7118 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: tablet copy: Starting download of 0 data blocks...
I20260929 11:21:15.781786 7118 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: tablet copy: Starting download of 1 WAL segments...
I20260929 11:21:15.783720 7118 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 11:21:15.786713 6578 tablet_copy_service.cc:342] P 08ec214716054f739cd17e77b4e5473d: Request end of tablet copy session c52b984fa50548ef81ac294b248bdc16-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36616
I20260929 11:21:15.786841 6578 tablet_copy_service.cc:433] P 08ec214716054f739cd17e77b4e5473d: ending tablet copy session c52b984fa50548ef81ac294b248bdc16-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c52b984fa50548ef81ac294b248bdc16
I20260929 11:21:15.797529 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:15.800047 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:15.809597 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:15.812120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6619
W20260929 11:21:15.812316 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:15.817920 7118 server_base.cc:1034] running on GCE node
I20260929 11:21:15.818291 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:15.819084 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:15.820338 7118 hybrid_clock.cc:648] HybridClock initialized: now 1790680875820286 us; error 57 us; skew 500 ppm
I20260929 11:21:15.820722 7118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:15 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:15.822952 7118 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 11:21:15.826907 7118 webserver.cc:460] Webserver started at http://127.0.97.60:36095/ using document root <none> and password file <none>
I20260929 11:21:15.827258 7118 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:15.827355 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:15.831040 7118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:15.833173 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:15.833757 7118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 11:21:15.833884 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
uuid: "c52b984fa50548ef81ac294b248bdc16"
format_stamp: "Formatted at 2026-09-29 11:21:15 on dist-test-slave-srst"
server_key: "39902bc2c24fa467def6790d8d160be8"
server_key_iv: "b390c13a272bf3924bddadef786ad5bc"
server_key_version: "encryptionkey@0"
I20260929 11:21:15.834069 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:15.835201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35101
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33079
--webserver_port=34513
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:21:15.861636 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:15.865532 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:15.904321 7118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46339
I20260929 11:21:15.904513 7391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46339 every 8 connection(s)
I20260929 11:21:15.905704 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
I20260929 11:21:15.911309 7392 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:21:15.915313 7392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Bootstrap starting.
I20260929 11:21:15.931247 7392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:21:15.931789 7392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Bootstrap complete.
I20260929 11:21:15.932868 7392 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } }
I20260929 11:21:15.933157 7392 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c52b984fa50548ef81ac294b248bdc16, State: Initialized, Role: LEARNER
I20260929 11:21:15.933399 7392 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } }
I20260929 11:21:15.934198 7395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:15.934571 7392 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:15.934718 7395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:21:15.945632 7406 catalog_manager.cc:1259] Loaded cluster ID: 4d275977890e45628abadcad783f75db
I20260929 11:21:15.945734 7406 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: loading cluster ID for follower catalog manager: success
I20260929 11:21:15.948926 7406 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: acquiring CA information for follower catalog manager: success
I20260929 11:21:15.951141 7406 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:21:16.168542 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:16.168895 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:16.175890 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.168975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.0.97.60:46339)
0929 11:21:16.169661 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.169666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.169683 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:21:16.169739 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.169742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.169829 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.169937 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:16.170362 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.171250 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.174093 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.174842 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.174952 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.175184 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.175494 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.175606 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":462,"threads_started":1}
I20260929 11:21:16.175921 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.168892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.0.1:53864)
0929 11:21:16.169393 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.169418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.169586 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.169987 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.169994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.170049 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:16.170228 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.170237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.171390 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.171397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.173915 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.173930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.175060 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.175067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.175275 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.175316 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.175425 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.175582 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":296,"threads_started":1}
I20260929 11:21:16.176872 7367 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260929 11:21:16.177368 7395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ec214716054f739cd17e77b4e5473d. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: true } } }
I20260929 11:21:16.177671 7395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:21:16.178056 7285 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: attempting to promote NON_VOTER c52b984fa50548ef81ac294b248bdc16 to VOTER
I20260929 11:21:16.178755 7285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } }
I20260929 11:21:16.180546 7367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 LEARNER]: Refusing update from remote peer 08ec214716054f739cd17e77b4e5473d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 11:21:16.180631 7172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Refusing update from remote peer 08ec214716054f739cd17e77b4e5473d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 11:21:16.181200 7285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 11:21:16.181669 7284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 11:21:16.184396 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c52b984fa50548ef81ac294b248bdc16 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.185014 7172 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c52b984fa50548ef81ac294b248bdc16 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.187698 7367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c52b984fa50548ef81ac294b248bdc16 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.188148 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.188416 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:16.188665 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.188910 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:16.189136 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.189340 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:16.192790 7395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.193112 7395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:16.193499 7410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08ec214716054f739cd17e77b4e5473d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:16.193778 7410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 11:21:16.210196 7344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:16.210446 7344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:16.210539 7344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:16.218027 7344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:16.218133 7344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:16.218251 7344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:16.228453 7344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34513
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:16.229835 7344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:16.230993 7344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:16.245043 7416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:16.245069 7419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:16.245106 7417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:16.245985 7344 server_base.cc:1034] running on GCE node
I20260929 11:21:16.246565 7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:16.247434 7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:16.248677 7344 hybrid_clock.cc:648] HybridClock initialized: now 1790680876248614 us; error 59 us; skew 500 ppm
I20260929 11:21:16.249006 7344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:16.251883 7344 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 11:21:16.253450 7344 webserver.cc:460] Webserver started at http://127.0.97.1:34513/ using document root <none> and password file <none>
I20260929 11:21:16.254077 7344 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:16.254220 7344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:16.259433 7344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 11:21:16.262193 7426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:16.263594 7344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:21:16.263753 7344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "b67946dde31c42c6bc1f9abc3aa03034"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "f54d164f5d6aea48d6840a7aeb97d6a1"
server_key_iv: "25f013b59789280398321afedf9a852e"
server_key_version: "encryptionkey@0"
I20260929 11:21:16.264689 7344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:16.280710 7344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:16.284389 7344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:16.284725 7344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:16.286036 7344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:16.287299 7344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:16.287417 7344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:16.287556 7344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:16.287649 7344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:16.322293 7344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35101
I20260929 11:21:16.322347 7538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35101 every 8 connection(s)
I20260929 11:21:16.323712 7344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:16.324910 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 7344
I20260929 11:21:16.325258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6753
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 11:21:16.369158 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.328026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39439 (local address 127.0.97.60:46339)
0929 11:21:16.328249 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.328257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.345354 (+ 17097us) server_negotiation.cc:407] Connection header received
0929 11:21:16.350988 (+ 5634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.350998 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.351080 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.351147 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.353226 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.354231 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.356067 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.356704 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.356786 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.362540 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.362547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.362551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.362593 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.365807 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366632 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.366637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.366640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.366783 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.367205 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.367209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.367212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.367555 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.367667 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.368552 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.368805 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:21:16.369304 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.326690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46283 (local address 127.0.97.62:33079)
0929 11:21:16.339462 (+ 12772us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.339471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.345942 (+ 6471us) server_negotiation.cc:407] Connection header received
0929 11:21:16.348130 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.348135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.348210 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.348302 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.351049 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.352157 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.355080 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.355690 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.355817 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.361147 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.361154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.361160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.361202 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.365456 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366430 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.366435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.366438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.366594 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.367056 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.367061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.367064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.367825 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.367959 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.368549 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.368884 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12589,"thread_start_us":12803,"threads_started":1}
I20260929 11:21:16.369947 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.326711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.97.2:40713)
0929 11:21:16.346690 (+ 19979us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.346699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.347433 (+ 734us) server_negotiation.cc:407] Connection header received
0929 11:21:16.347526 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.347534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.347795 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.347945 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.350273 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.351791 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.353757 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.354569 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.354683 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.361231 (+ 6548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.361245 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.361264 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.361299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.365595 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366439 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.366448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.366451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.366594 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.367048 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.367052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.367055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.367868 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.368136 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.369252 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.369600 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19809,"thread_start_us":19672,"threads_started":1}
I20260929 11:21:16.371176 6876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:16.373798 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.346367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55503 (local address 127.0.97.61:35157)
0929 11:21:16.350605 (+ 4238us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.350613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.351057 (+ 444us) server_negotiation.cc:407] Connection header received
0929 11:21:16.351119 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.351126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.351194 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.351851 (+ 657us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.353033 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.354120 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.356581 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.357224 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.357332 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.361759 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.361764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.361769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.361804 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.365258 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366357 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.366362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.366365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.366509 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.367002 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.367008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.367011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.367396 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.367533 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.369000 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.369201 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4066,"thread_start_us":3942,"threads_started":1}
I20260929 11:21:16.375952 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.327303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.97.1:39439)
0929 11:21:16.345157 (+ 17854us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.345208 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.350791 (+ 5583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.351338 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.351342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.352166 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:16.353050 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.353063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.354395 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.354404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.355908 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.355921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.357061 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.357067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.357238 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.357828 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:16.357848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:16.362328 (+ 4480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:16.365987 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.365992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.365996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.366509 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.366931 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.366940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.367085 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.367752 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:16.367757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:16.368143 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.369382 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.374770 (+ 5388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17542,"mutex_wait_us":5249,"spinlock_wait_cycles":22272,"thread_start_us":18612,"threads_started":1}
I20260929 11:21:16.375972 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.347293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.97.1:55503)
0929 11:21:16.350833 (+ 3540us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.350851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.350962 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.351338 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.351342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.351885 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:16.352803 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.352813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.354281 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.354287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.356428 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.356442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.357621 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.357625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.357872 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.358389 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:16.358408 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:16.361584 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:16.365446 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.365456 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.365475 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.366203 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.366627 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.366635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.366881 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.367604 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:16.367615 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:16.368107 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.368435 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.374425 (+ 5990us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3414,"mutex_wait_us":562,"spinlock_wait_cycles":19712,"thread_start_us":3315,"threads_started":1}
I20260929 11:21:16.376576 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.328463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:46283)
0929 11:21:16.345832 (+ 17369us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.345849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.347803 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.348930 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.348941 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.349932 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:16.350840 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.350863 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.352667 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.352673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.354943 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.354953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.356134 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.356143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.356453 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.357206 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:16.357239 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:16.360810 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:16.365613 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.365619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.365625 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.366285 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.366738 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.366743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.366746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.366917 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.368051 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:16.368055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:16.368201 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.369581 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.376305 (+ 6724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17253,"mutex_wait_us":6502,"spinlock_wait_cycles":4352,"thread_start_us":16312,"threads_started":1}
I20260929 11:21:16.378271 7541 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35157
I20260929 11:21:16.378414 7539 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46339
I20260929 11:21:16.378605 7539 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:16.378608 7541 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:16.378985 7540 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260929 11:21:16.379102 7540 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:16.379621 7541 heartbeater.cc:511] Master 127.0.97.61:35157 requested a full tablet report, sending...
I20260929 11:21:16.379649 7540 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260929 11:21:16.380363 7539 heartbeater.cc:511] Master 127.0.97.60:46339 requested a full tablet report, sending...
I20260929 11:21:16.381330 6558 ts_manager.cc:194] Re-registered known tserver with Master: b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101)
I20260929 11:21:16.382143 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46283
I20260929 11:21:16.382280 7357 ts_manager.cc:194] Registered new tserver with Master: b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101)
I20260929 11:21:16.382280 7162 ts_manager.cc:194] Registered new tserver with Master: b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101)
I20260929 11:21:16.457566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6753
I20260929 11:21:16.473767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40713
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33079
--webserver_port=45705
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:16.792690 7551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:16.792959 7551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:16.793062 7551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:16.800213 7551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:16.800309 7551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:16.800415 7551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:16.810153 7551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45705
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:16.811407 7551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:16.812680 7551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:16.825888 7556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:16.825827 7557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:16.825914 7559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:16.826947 7551 server_base.cc:1034] running on GCE node
I20260929 11:21:16.827534 7551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:16.828423 7551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:16.829636 7551 hybrid_clock.cc:648] HybridClock initialized: now 1790680876829582 us; error 45 us; skew 500 ppm
I20260929 11:21:16.829936 7551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:16.832911 7551 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 11:21:16.834232 7551 webserver.cc:460] Webserver started at http://127.0.97.2:45705/ using document root <none> and password file <none>
I20260929 11:21:16.834968 7551 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:16.835100 7551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:16.840489 7551 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:21:16.843252 7566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:16.844506 7551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:21:16.844686 7551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "8cf6600aabbe4cfea93c14e272f75a14"
format_stamp: "Formatted at 2026-09-29 11:21:13 on dist-test-slave-srst"
server_key: "6670d3de481b43a85b0b3b99ef9edf59"
server_key_iv: "c3717fe5d4e8b71d6d3d332b22f61ae1"
server_key_version: "encryptionkey@0"
I20260929 11:21:16.845613 7551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:16.861055 7551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:16.865198 7551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:16.865453 7551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:16.866353 7551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:16.867759 7551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:16.867870 7551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:16.867980 7551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:16.868078 7551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:16.898252 7551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40713
I20260929 11:21:16.898296 7678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40713 every 8 connection(s)
I20260929 11:21:16.899636 7551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:16.902911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 7551
I20260929 11:21:16.903241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6886
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 11:21:16.930112 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.904139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.97.3:37223)
0929 11:21:16.909128 (+ 4989us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.909137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.909165 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:21:16.909248 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.909256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.909478 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.911589 (+ 2111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.911834 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.913398 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.915278 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.916107 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.916231 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.922264 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.922277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.922294 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.922338 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.926231 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.927053 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.927061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.927064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.927211 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.927678 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.927683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.927686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.928074 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.928343 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.929414 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.929687 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4823,"thread_start_us":4727,"threads_started":1}
I20260929 11:21:16.931671 7009 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 11:21:16 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:16.948695 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.902292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33023 (local address 127.0.97.61:35157)
0929 11:21:16.903895 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.903901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.921120 (+ 17219us) server_negotiation.cc:407] Connection header received
0929 11:21:16.925602 (+ 4482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.925611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.925695 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.925777 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.928267 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.929321 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.935407 (+ 6086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.935984 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.936100 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.940907 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.940913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.940918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.940968 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.944148 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.945782 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.945789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.945794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.945975 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.946545 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.946550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.946554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.946968 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.947102 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.948163 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.948376 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1475,"thread_start_us":6361,"threads_started":1}
I20260929 11:21:16.951001 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.902982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.97.2:33023)
0929 11:21:16.920957 (+ 17975us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.921019 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.924726 (+ 3707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.926227 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.926238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.927216 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:16.928081 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.928106 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.929534 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.929538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.931648 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.931665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.936374 (+ 4709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.936379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.936561 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.937403 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:16.937435 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:16.940672 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:16.944370 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.944381 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.944399 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.945624 (+ 1225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.946109 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.946114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.946119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.946369 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.947193 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:16.947204 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:16.947694 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.948999 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.949471 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17592,"thread_start_us":18182,"threads_started":1}
I20260929 11:21:16.952997 7681 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35157
I20260929 11:21:16.953147 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.903248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45105 (local address 127.0.97.60:46339)
0929 11:21:16.925136 (+ 21888us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.925142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.925160 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:21:16.925212 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.925218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.925299 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.925399 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.933082 (+ 7683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.934602 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.938308 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.938962 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.939077 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.944843 (+ 5766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.944851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.944858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.944905 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.949788 (+ 4883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.950638 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.950643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.950647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.950779 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.951186 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.951191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.951194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.951894 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.952019 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.952523 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.952773 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21746,"thread_start_us":23417,"threads_started":1}
I20260929 11:21:16.953316 7681 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:16.954133 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.902982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.97.2:45105)
0929 11:21:16.921617 (+ 18635us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.921637 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.924293 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.931817 (+ 7524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.931821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.932415 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:16.932813 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.932829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.934806 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.934814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.938127 (+ 3313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.938141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.939357 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.939363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.939555 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.940767 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:16.940789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:16.944578 (+ 3789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:16.949938 (+ 5360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.949943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.950009 (+ 66us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.950478 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.950904 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.950909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.950913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.951065 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.952096 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:16.952101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:16.952258 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.953380 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.953700 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18477,"mutex_wait_us":389,"thread_start_us":17013,"threads_started":1}
I20260929 11:21:16.954336 7681 heartbeater.cc:511] Master 127.0.97.61:35157 requested a full tablet report, sending...
I20260929 11:21:16.954888 7679 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46339
I20260929 11:21:16.955003 7679 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:16.955246 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.921955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52769 (local address 127.0.97.62:33079)
0929 11:21:16.931864 (+ 9909us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:16.931873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:16.937468 (+ 5595us) server_negotiation.cc:407] Connection header received
0929 11:21:16.937632 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:16.937639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:16.937717 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:16.937825 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:16.938962 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.940144 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.942060 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.942765 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.942898 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.948764 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:16.948770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:16.948775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:16.948812 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:16.952315 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.953072 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.953076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.953079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.953217 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:16.953718 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:16.953724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:16.953726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:16.954062 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:16.954188 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:16.954681 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 11:21:16.954885 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9746,"thread_start_us":9919,"threads_started":1}
I20260929 11:21:16.955363 7679 heartbeater.cc:511] Master 127.0.97.60:46339 requested a full tablet report, sending...
I20260929 11:21:16.956055 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:16.921797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:52769)
0929 11:21:16.937367 (+ 15570us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:16.937386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:16.937515 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:16.937872 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:16.937879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:16.938445 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:16.938798 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.938814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.940282 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.940299 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.941926 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:16.941942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:16.942961 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:16.942968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:16.943163 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:16.944424 (+ 1261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:16.944482 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:16.947991 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:16.952464 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.952468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.952472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.952929 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.953389 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:16.953393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:16.953397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:16.953585 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:16.954263 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:16.954268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:16.954398 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 11:21:16.955403 (+ 1005us) client_negotiation.cc:240] Negotiation successful
0929 11:21:16.955696 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15097,"thread_start_us":14950,"threads_started":1}
I20260929 11:21:16.956488 7357 ts_manager.cc:194] Registered new tserver with Master: 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713)
I20260929 11:21:16.956591 7162 ts_manager.cc:194] Registered new tserver with Master: 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713)
I20260929 11:21:16.957855 7680 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260929 11:21:16.957990 7680 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:16.958345 7680 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260929 11:21:16.959518 6558 ts_manager.cc:194] Re-registered known tserver with Master: 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713)
I20260929 11:21:16.960391 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52769
I20260929 11:21:17.021974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6886
I20260929 11:21:17.038070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37223
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33079
--webserver_port=35835
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:17.350338 7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:17.350608 7692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:17.350751 7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:17.357910 7692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:17.358004 7692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:17.358166 7692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:17.367033 7692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35835
--tserver_master_addrs=127.0.97.62:33079,127.0.97.61:35157,127.0.97.60:46339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:17.368347 7692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:17.369695 7692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:17.381930 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:17.381901 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:17.382133 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:17.382623 7692 server_base.cc:1034] running on GCE node
I20260929 11:21:17.383198 7692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:17.384271 7692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:17.385393 7540 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
I20260929 11:21:17.385491 7692 hybrid_clock.cc:648] HybridClock initialized: now 1790680877385439 us; error 39 us; skew 500 ppm
I20260929 11:21:17.385861 7692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:17 dist-test-slave-srst krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:17.389034 7692 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 11:21:17.390419 7692 webserver.cc:460] Webserver started at http://127.0.97.3:35835/ using document root <none> and password file <none>
I20260929 11:21:17.391153 7692 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:17.391306 7692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:17.396456 7692 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:17.399463 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:17.400890 7692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:21:17.401067 7692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "4855b9133cb94dc8970dad7192a7e497"
format_stamp: "Formatted at 2026-09-29 11:21:14 on dist-test-slave-srst"
server_key: "cd3685a22e9b8ca36ea7f4c2f4730920"
server_key_iv: "68819b7ac0a9c31189828e5363ab330d"
server_key_version: "encryptionkey@0"
I20260929 11:21:17.402201 7692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:17.422793 7692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:17.426952 7692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:17.427278 7692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:17.428283 7692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:17.429740 7692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:17.429869 7692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:17.430023 7692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:17.430115 7692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:17.471957 7692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37223
I20260929 11:21:17.472000 7819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37223 every 8 connection(s)
I20260929 11:21:17.473450 7692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:17.476891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 7692
Sep 29 11:21:17 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 11:21:17 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 11:21:17.517593 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.477673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57867 (local address 127.0.97.60:46339)
0929 11:21:17.482136 (+ 4463us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.482145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.482170 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:21:17.482248 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.482256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.482370 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.482497 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.485332 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.486826 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.495671 (+ 8845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.496737 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.497473 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.505601 (+ 8128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.505612 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.505620 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.505676 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.510482 (+ 4806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.512988 (+ 2506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.512995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.513001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.513162 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.513734 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.513741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.513747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.514153 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.514304 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.516803 (+ 2499us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.517077 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4271,"thread_start_us":4121,"threads_started":1}
I20260929 11:21:17.519048 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.477079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.97.3:57867)
0929 11:21:17.478051 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.478106 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.481291 (+ 3185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.483070 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.483086 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.484061 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:17.485072 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.485105 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.487029 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.487036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.495407 (+ 8371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.495446 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.498302 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.498311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.498708 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.500035 (+ 1327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:17.500074 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:17.505239 (+ 5165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:17.510685 (+ 5446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:17.510699 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:17.510723 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:17.512817 (+ 2094us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:17.513304 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:17.513312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:17.513318 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:17.513578 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:17.514379 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:17.514390 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:17.514905 (+ 515us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.515229 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.515861 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":2,"thread_start_us":885,"threads_started":1}
I20260929 11:21:17.525429 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.476354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50481 (local address 127.0.97.61:35157)
0929 11:21:17.476824 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.476830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.487511 (+ 10681us) server_negotiation.cc:407] Connection header received
0929 11:21:17.487674 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.487680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.487762 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.487878 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.489885 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.491354 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.498124 (+ 6770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.499030 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.499185 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.513316 (+ 14131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.513326 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.513334 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.513400 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.519843 (+ 6443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.521129 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.521137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.521143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.521321 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.521837 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.521844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.521850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.523609 (+ 1759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.523769 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.524490 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.524911 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":257,"threads_started":1}
I20260929 11:21:17.527099 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.477849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.97.3:50481)
0929 11:21:17.487395 (+ 9546us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.487422 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.487552 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.488431 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.488438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.489289 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:17.489706 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.489721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.491499 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.491506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.497784 (+ 6278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.497803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.499301 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.499310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.499552 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.500714 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:17.500733 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:17.513024 (+ 12291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:17.520035 (+ 7011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:17.520043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:17.520049 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:17.520948 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:17.521486 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:17.521493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:17.521496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:17.521693 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:17.523895 (+ 2202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:17.523902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:17.524119 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.526049 (+ 1930us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.526446 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9406,"mutex_wait_us":2480,"thread_start_us":9185,"threads_started":1}
I20260929 11:21:17.529407 7822 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35157
I20260929 11:21:17.529781 7822 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:17.530885 7822 heartbeater.cc:511] Master 127.0.97.61:35157 requested a full tablet report, sending...
Sep 29 11:21:17 dist-test-slave-srst krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:17.532872 7820 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46339
I20260929 11:21:17.532999 7820 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:17.533438 7820 heartbeater.cc:511] Master 127.0.97.60:46339 requested a full tablet report, sending...
I20260929 11:21:17.533705 7162 ts_manager.cc:194] Registered new tserver with Master: 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:17.534662 7357 ts_manager.cc:194] Registered new tserver with Master: 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:17.544703 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.487735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:54367)
0929 11:21:17.502328 (+ 14593us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.502354 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.502501 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.502857 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.502861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.509666 (+ 6805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:17.517927 (+ 8261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.517952 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.519662 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.519670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.522034 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.522052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.524972 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.524981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.525221 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.526308 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:17.526336 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:17.534238 (+ 7902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:17.541669 (+ 7431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:17.541673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:17.541676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:17.542001 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:17.542433 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:17.542438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:17.542442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:17.542615 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:17.543560 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:17.543568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:17.543779 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.543916 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.544241 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13773,"thread_start_us":13430,"threads_started":1}
I20260929 11:21:17.546080 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.487772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54367 (local address 127.0.97.62:33079)
0929 11:21:17.500320 (+ 12548us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.500329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.502435 (+ 2106us) server_negotiation.cc:407] Connection header received
0929 11:21:17.502637 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.502644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.502714 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.502824 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.518130 (+ 15306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.519531 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.522185 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.523069 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.523237 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.534855 (+ 11618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.534861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.534865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.534924 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.539817 (+ 4893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.542125 (+ 2308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.542129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.542131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.542317 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.543064 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.543069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.543072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.543421 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.545329 (+ 1908us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.545454 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.545670 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12338,"thread_start_us":12559,"threads_started":1}
I20260929 11:21:17.547626 7821 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260929 11:21:17.547766 7821 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:17.548244 7821 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260929 11:21:17.549386 6558 ts_manager.cc:194] Re-registered known tserver with Master: 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:17.550104 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.503146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.0.97.62:33079)
0929 11:21:17.528641 (+ 25495us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.528649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.528671 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:21:17.528735 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.528742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.528860 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.528981 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.532314 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.533510 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.538084 (+ 4574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.538716 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.538811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.542758 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.542764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.542768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.542808 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.546372 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.547575 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.547582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.547591 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.547743 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.548151 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.548156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.548159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.548500 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.549202 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.549446 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.549687 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25307,"thread_start_us":29002,"threads_started":1}
I20260929 11:21:17.550527 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54367
I20260929 11:21:17.552961 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.503303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.0.97.60:46339)
0929 11:21:17.517852 (+ 14549us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.517856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.535023 (+ 17167us) server_negotiation.cc:407] Connection header received
0929 11:21:17.535249 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.535256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.535353 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.535402 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.538347 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.539337 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.542554 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.543130 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.543247 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.546906 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.546912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.546917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.546954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.550071 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.550979 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.550983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.550985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.551096 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.551500 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.551505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.551508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.551769 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.552013 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.552340 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.552541 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14436,"thread_start_us":28718,"threads_started":1}
I20260929 11:21:17.560922 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.535198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.0.97.61:35157)
0929 11:21:17.535380 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.535386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.542868 (+ 7482us) server_negotiation.cc:407] Connection header received
0929 11:21:17.543034 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.543040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.543147 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.543235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.545833 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.546946 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.551347 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.551936 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.552035 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.555661 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.555666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.555669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.555701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.558491 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.559087 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.559092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.559097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.559216 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.559590 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.559594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.559596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.559823 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.560070 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.560393 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.560637 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:21:17.555524 6557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:17.562832 6557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:17.608727 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.580883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.0.97.2:40713)
0929 11:21:17.600227 (+ 19344us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.600233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.600254 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:21:17.600323 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.600327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.600558 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.601001 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:17.601673 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.603196 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.606281 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.607740 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.607957 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.608108 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.608336 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.608455 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19204,"thread_start_us":19127,"threads_started":1}
I20260929 11:21:17.608834 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.580656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40713 (local address 127.0.0.1:52926)
0929 11:21:17.592908 (+ 12252us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.592933 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.594213 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.600816 (+ 6603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.600821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.600856 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:17.601464 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.601475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.603326 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.603332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.606087 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.606098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.607973 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.607979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.608157 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.608191 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.608284 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.608417 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12085,"mutex_wait_us":112,"thread_start_us":20336,"threads_started":1}
I20260929 11:21:17.609951 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.581022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35101 (local address 127.0.0.1:50194)
0929 11:21:17.599762 (+ 18740us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.599777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.599876 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.600528 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.600532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.600556 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:17.601070 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.601083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.603384 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.603394 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.606125 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.606138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.609185 (+ 3047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.609195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.609378 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.609406 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.609479 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.609587 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18603,"mutex_wait_us":83,"thread_start_us":18491,"threads_started":1}
I20260929 11:21:17.609948 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.581624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.0.97.1:35101)
0929 11:21:17.594060 (+ 12436us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.594069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.599896 (+ 5827us) server_negotiation.cc:407] Connection header received
0929 11:21:17.600036 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.600044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.600335 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.600501 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:17.601730 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.603232 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.606329 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.607744 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.607886 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.608099 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.609562 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.609681 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12249,"thread_start_us":12160,"threads_started":1}
I20260929 11:21:17.610585 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.581493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37223 (local address 127.0.0.1:38220)
0929 11:21:17.601224 (+ 19731us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.601243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.601361 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.601909 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.601914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.601934 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:17.602253 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.602261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.603968 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.603974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.606838 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.606848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.608793 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.608805 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.608992 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.609024 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.610157 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.610267 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19446,"thread_start_us":11106,"threads_started":1}
I20260929 11:21:17.610708 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.581919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.97.3:37223)
0929 11:21:17.583137 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.583146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.601350 (+ 18204us) server_negotiation.cc:407] Connection header received
0929 11:21:17.601484 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.601493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.601735 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.601910 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:17.602433 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.603624 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.606964 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.607964 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.608098 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.608348 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.610202 (+ 1854us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.610366 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":820,"threads_started":1}
I20260929 11:21:17.612820 7614 tablet_service.cc:1467] Processing CreateTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 (DEFAULT_TABLE table=test-table [id=d0116f1ebfa74b2ebf95fc2b0dc6b448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:17.613250 7474 tablet_service.cc:1467] Processing CreateTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 (DEFAULT_TABLE table=test-table [id=d0116f1ebfa74b2ebf95fc2b0dc6b448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:17.614046 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25a2d9d340c45fa80b6bc0a49aa7253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:17.614377 7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25a2d9d340c45fa80b6bc0a49aa7253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:17.620980 7755 tablet_service.cc:1467] Processing CreateTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 (DEFAULT_TABLE table=test-table [id=d0116f1ebfa74b2ebf95fc2b0dc6b448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:17.622400 7755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25a2d9d340c45fa80b6bc0a49aa7253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:17.625851 7857 tablet_bootstrap.cc:492] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Bootstrap starting.
I20260929 11:21:17.629285 7858 tablet_bootstrap.cc:492] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Bootstrap starting.
I20260929 11:21:17.629328 7857 tablet_bootstrap.cc:654] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:17.630893 7857 log.cc:824] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:17.632699 7858 tablet_bootstrap.cc:654] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:17.633816 7858 log.cc:824] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:17.634155 7857 tablet_bootstrap.cc:492] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: No bootstrap required, opened a new log
I20260929 11:21:17.634512 7857 ts_tablet_manager.cc:1397] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:21:17.637054 7861 tablet_bootstrap.cc:492] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Bootstrap starting.
I20260929 11:21:17.639994 7858 tablet_bootstrap.cc:492] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: No bootstrap required, opened a new log
I20260929 11:21:17.640026 7861 tablet_bootstrap.cc:654] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:17.640300 7858 ts_tablet_manager.cc:1397] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260929 11:21:17.641139 7861 log.cc:824] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:17.644418 7861 tablet_bootstrap.cc:492] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: No bootstrap required, opened a new log
I20260929 11:21:17.644722 7861 ts_tablet_manager.cc:1397] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260929 11:21:17.646981 7857 raft_consensus.cc:348] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.647411 7857 raft_consensus.cc:374] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:17.647542 7857 raft_consensus.cc:729] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67946dde31c42c6bc1f9abc3aa03034, State: Initialized, Role: FOLLOWER
I20260929 11:21:17.648154 7857 consensus_queue.cc:260] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.650416 7857 ts_tablet_manager.cc:1428] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 11:21:17.651391 7858 raft_consensus.cc:348] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.651880 7858 raft_consensus.cc:374] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:17.652020 7858 raft_consensus.cc:729] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf6600aabbe4cfea93c14e272f75a14, State: Initialized, Role: FOLLOWER
I20260929 11:21:17.652763 7858 consensus_queue.cc:260] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.653685 7861 raft_consensus.cc:348] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.654145 7861 raft_consensus.cc:374] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:17.654276 7861 raft_consensus.cc:729] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4855b9133cb94dc8970dad7192a7e497, State: Initialized, Role: FOLLOWER
I20260929 11:21:17.654918 7861 consensus_queue.cc:260] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.657470 7858 ts_tablet_manager.cc:1428] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260929 11:21:17.657855 7680 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
I20260929 11:21:17.661898 7861 ts_tablet_manager.cc:1428] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260929 11:21:17.662199 7821 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
W20260929 11:21:17.726477 7823 tablet.cc:2369] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:17.807715 7864 raft_consensus.cc:482] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:17.808051 7864 raft_consensus.cc:504] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.809877 7864 leader_election.cc:288] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101), 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:17.816723 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.810360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51691 (local address 127.0.97.3:37223)
0929 11:21:17.810581 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.810589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.811647 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 11:21:17.811809 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.811812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.811864 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.811948 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:17.812541 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.813272 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.815102 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.815800 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.815966 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.816069 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.816316 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.816433 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:21:17.816810 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.810140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37223 (local address 127.0.97.2:51691)
0929 11:21:17.811524 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.811537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.811681 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.811963 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.811966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.812050 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:17.812366 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.812374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.813399 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.813405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.814952 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.814963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.816004 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.816009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.816156 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.816183 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.816282 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.816399 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"mutex_wait_us":22,"spinlock_wait_cycles":23296,"thread_start_us":462,"threads_started":1}
I20260929 11:21:17.816987 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.810140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49175 (local address 127.0.97.1:35101)
0929 11:21:17.810402 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.810411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.811603 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 11:21:17.811808 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.811812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.811863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.811949 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:17.812497 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.813268 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.815124 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.815933 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.816051 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.816122 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.816549 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.816741 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:21:17.817600 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.810057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35101 (local address 127.0.97.2:49175)
0929 11:21:17.811485 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:17.811506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:17.811660 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:17.811962 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:17.811966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:17.811987 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:17.812366 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.812374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.813403 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.813408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.814960 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:17.814971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.816234 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.816238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:17.816389 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.816417 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:21:17.817172 (+ 755us) client_negotiation.cc:240] Negotiation successful
0929 11:21:17.817304 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":75,"thread_start_us":1244,"threads_started":1}
I20260929 11:21:17.818101 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25a2d9d340c45fa80b6bc0a49aa7253" candidate_uuid: "8cf6600aabbe4cfea93c14e272f75a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4855b9133cb94dc8970dad7192a7e497" is_pre_election: true
I20260929 11:21:17.818571 7494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25a2d9d340c45fa80b6bc0a49aa7253" candidate_uuid: "8cf6600aabbe4cfea93c14e272f75a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67946dde31c42c6bc1f9abc3aa03034" is_pre_election: true
I20260929 11:21:17.818637 7775 raft_consensus.cc:2454] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cf6600aabbe4cfea93c14e272f75a14 in term 0.
I20260929 11:21:17.819016 7494 raft_consensus.cc:2454] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cf6600aabbe4cfea93c14e272f75a14 in term 0.
I20260929 11:21:17.819336 7568 leader_election.cc:302] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4855b9133cb94dc8970dad7192a7e497, 8cf6600aabbe4cfea93c14e272f75a14; no voters:
I20260929 11:21:17.819754 7864 raft_consensus.cc:2781] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:17.819902 7864 raft_consensus.cc:482] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:17.819999 7864 raft_consensus.cc:3037] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:17.822806 7864 raft_consensus.cc:504] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.823446 7864 leader_election.cc:288] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [CANDIDATE]: Term 1 election: Requested vote from peers b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101), 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:17.823930 7494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25a2d9d340c45fa80b6bc0a49aa7253" candidate_uuid: "8cf6600aabbe4cfea93c14e272f75a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67946dde31c42c6bc1f9abc3aa03034"
I20260929 11:21:17.823971 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25a2d9d340c45fa80b6bc0a49aa7253" candidate_uuid: "8cf6600aabbe4cfea93c14e272f75a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4855b9133cb94dc8970dad7192a7e497"
I20260929 11:21:17.824178 7494 raft_consensus.cc:3037] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:17.824208 7775 raft_consensus.cc:3037] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:17.827474 7775 raft_consensus.cc:2454] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cf6600aabbe4cfea93c14e272f75a14 in term 1.
I20260929 11:21:17.827474 7494 raft_consensus.cc:2454] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cf6600aabbe4cfea93c14e272f75a14 in term 1.
I20260929 11:21:17.827886 7568 leader_election.cc:302] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4855b9133cb94dc8970dad7192a7e497, 8cf6600aabbe4cfea93c14e272f75a14; no voters:
I20260929 11:21:17.828210 7864 raft_consensus.cc:2781] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:17.828743 7864 raft_consensus.cc:686] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 LEADER]: Becoming Leader. State: Replica: 8cf6600aabbe4cfea93c14e272f75a14, State: Running, Role: LEADER
I20260929 11:21:17.829303 7864 consensus_queue.cc:237] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:17.834982 6557 catalog_manager.cc:5495] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8cf6600aabbe4cfea93c14e272f75a14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } health_report { overall_health: UNKNOWN } } }
W20260929 11:21:17.841976 7547 tablet.cc:2369] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:17.885980 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.880551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.97.2:40713)
0929 11:21:17.880763 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.880767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.880784 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:21:17.880867 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.880871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.880949 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.881069 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:17.881611 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.882679 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.883984 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.884367 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.884459 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.884954 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:17.885445 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:17.885574 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.885610 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.885668 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:21:17.894871 7494 raft_consensus.cc:1261] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Refusing update from remote peer 8cf6600aabbe4cfea93c14e272f75a14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:17.894871 7775 raft_consensus.cc:1261] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Refusing update from remote peer 8cf6600aabbe4cfea93c14e272f75a14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:17.895845 7868 consensus_queue.cc:1035] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:17.896207 7864 consensus_queue.cc:1035] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:21:17.903016 7682 tablet.cc:2369] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:17.937749 6557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:17.938014 6557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:17.947471 6557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08ec214716054f739cd17e77b4e5473d: Sending DeleteTablet for 3 replicas of tablet a25a2d9d340c45fa80b6bc0a49aa7253
I20260929 11:21:17.948433 7755 tablet_service.cc:1514] Processing DeleteTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38220
I20260929 11:21:17.948894 7614 tablet_service.cc:1514] Processing DeleteTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52926
I20260929 11:21:17.949527 7474 tablet_service.cc:1514] Processing DeleteTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50194
I20260929 11:21:17.949759 7754 tablet_service.cc:1514] Processing DeleteTablet for tablet a25a2d9d340c45fa80b6bc0a49aa7253 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38220
I20260929 11:21:17.950824 7880 tablet_replica.cc:330] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: stopping tablet replica
I20260929 11:21:17.951303 7880 raft_consensus.cc:2229] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:17.951821 7880 raft_consensus.cc:2258] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:17.954237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 6526
I20260929 11:21:17.954517 7880 ts_tablet_manager.cc:1905] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:17.955423 6588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:17.965171 7882 tablet_replica.cc:330] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: stopping tablet replica
I20260929 11:21:17.965842 7882 raft_consensus.cc:2229] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:17.966503 7881 tablet_replica.cc:330] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: stopping tablet replica
I20260929 11:21:17.966883 7881 raft_consensus.cc:2229] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:17.967460 7882 raft_consensus.cc:2258] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:17.968379 7881 raft_consensus.cc:2258] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:17.970175 7882 ts_tablet_manager.cc:1905] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:17.971903 7881 ts_tablet_manager.cc:1905] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:17.972390 7880 ts_tablet_manager.cc:1918] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:17.973732 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.952113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.97.61:35157)
0929 11:21:17.952532 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.952536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.953476 (+ 940us) server_negotiation.cc:407] Connection header received
0929 11:21:17.953760 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.953765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.953843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.953917 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.956501 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.958069 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.960619 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.961427 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.961521 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.965701 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.965711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.965719 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.965772 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.969909 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.971012 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.971017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.971021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.971182 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.971621 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.971626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.971630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.971959 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.972195 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.972928 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.973226 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260929 11:21:17.976992 7880 log.cc:1196] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/a25a2d9d340c45fa80b6bc0a49aa7253
I20260929 11:21:17.977819 7880 ts_tablet_manager.cc:1939] T a25a2d9d340c45fa80b6bc0a49aa7253 P 4855b9133cb94dc8970dad7192a7e497: Deleting consensus metadata
I20260929 11:21:17.982847 7881 ts_tablet_manager.cc:1918] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:17.983057 7881 log.cc:1196] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/a25a2d9d340c45fa80b6bc0a49aa7253
I20260929 11:21:17.983704 7881 ts_tablet_manager.cc:1939] T a25a2d9d340c45fa80b6bc0a49aa7253 P 8cf6600aabbe4cfea93c14e272f75a14: Deleting consensus metadata
I20260929 11:21:17.986043 7882 ts_tablet_manager.cc:1918] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:17.986193 7882 log.cc:1196] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/a25a2d9d340c45fa80b6bc0a49aa7253
I20260929 11:21:17.986714 7882 ts_tablet_manager.cc:1939] T a25a2d9d340c45fa80b6bc0a49aa7253 P b67946dde31c42c6bc1f9abc3aa03034: Deleting consensus metadata
I20260929 11:21:17.992292 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:17.953491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.0.97.60:46339)
0929 11:21:17.953683 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:17.953689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:17.976745 (+ 23056us) server_negotiation.cc:407] Connection header received
0929 11:21:17.976989 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:17.976995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:17.977083 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:17.977183 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:17.979032 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.980111 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.982135 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:17.982791 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:17.982894 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:17.986784 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:17.986790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:17.986795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:17.986838 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:17.989536 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.990182 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.990188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.990191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.990341 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:17.990810 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:17.990815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:17.990818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:17.991118 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:17.991366 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:17.991705 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 11:21:17.991902 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:21:18.133780 6542 catalog_manager.cc:4841] TS 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713): tablet a25a2d9d340c45fa80b6bc0a49aa7253 (table test-table [id=d0116f1ebfa74b2ebf95fc2b0dc6b448]) successfully deleted
I20260929 11:21:18.133806 6543 catalog_manager.cc:4841] TS 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223): tablet a25a2d9d340c45fa80b6bc0a49aa7253 (table test-table [id=d0116f1ebfa74b2ebf95fc2b0dc6b448]) successfully deleted
I20260929 11:21:18.134064 6545 catalog_manager.cc:4841] TS b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101): tablet a25a2d9d340c45fa80b6bc0a49aa7253 (table test-table [id=d0116f1ebfa74b2ebf95fc2b0dc6b448]) successfully deleted
W20260929 11:21:18.134905 6543 catalog_manager.cc:4808] TS 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223): delete failed for tablet a25a2d9d340c45fa80b6bc0a49aa7253 because the tablet was not found. No further retry: Not found: Tablet not found: a25a2d9d340c45fa80b6bc0a49aa7253
I20260929 11:21:18.135602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 6526
I20260929 11:21:18.178578 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:18.161422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.0.97.60:46339)
0929 11:21:18.161680 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:18.161689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:18.162582 (+ 893us) server_negotiation.cc:407] Connection header received
0929 11:21:18.163154 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:18.163160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:18.163251 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:18.163331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:18.165525 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:18.166637 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:18.168583 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:18.169240 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:18.169365 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:18.171812 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:18.171818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:18.171822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:18.171857 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:18.175145 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:18.175915 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:18.175921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:18.175926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:18.176057 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:18.176446 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:18.176451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:18.176457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:18.176983 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:18.177227 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:18.177996 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 11:21:18.178227 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:21:18.178957 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:18.162381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.97.61:35157)
0929 11:21:18.162587 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:18.162593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:18.163404 (+ 811us) server_negotiation.cc:407] Connection header received
0929 11:21:18.163527 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:18.163532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:18.163610 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:18.163716 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:18.165813 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:18.166539 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:18.167974 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:18.168539 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:18.168744 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:18.172301 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:18.172323 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:18.172328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:18.172392 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:18.175560 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:18.176383 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:18.176390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:18.176393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:18.176539 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:18.176992 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:18.176996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:18.176999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:18.177329 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:18.177564 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:18.178441 (+ 877us) server_negotiation.cc:299] Negotiation successful
0929 11:21:18.178642 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:21:18.933241 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:18.932610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:59225)
0929 11:21:18.932991 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:18.933093 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":202,"threads_started":1}
W20260929 11:21:18.933625 7680 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:18.949887 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:18.949100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:42885)
0929 11:21:18.949487 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:18.949617 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":183,"threads_started":1}
W20260929 11:21:18.950305 7821 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:18.951491 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:18.950909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:53325)
0929 11:21:18.951247 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:18.951343 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":170,"threads_started":1}
W20260929 11:21:18.951845 7540 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:19.447309 7894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08ec214716054f739cd17e77b4e5473d)
I20260929 11:21:19.447537 7895 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08ec214716054f739cd17e77b4e5473d)
I20260929 11:21:19.447599 7894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } }
I20260929 11:21:19.447739 7895 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } }
I20260929 11:21:19.448498 7895 leader_election.cc:288] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079), 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61:35157)
I20260929 11:21:19.450006 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.448750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37022)
0929 11:21:19.449354 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.449551 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":273,"threads_started":1}
I20260929 11:21:19.461392 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.449340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37038)
0929 11:21:19.460850 (+ 11510us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.461081 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11390,"thread_start_us":11271,"threads_started":1}
I20260929 11:21:19.462391 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.449414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35157 (local address 127.0.0.1:57578)
0929 11:21:19.450262 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.450285 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.450447 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.451977 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.451985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.452026 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.452332 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.452346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.454608 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.454618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.457439 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.457457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.461620 (+ 4163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.461627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.461826 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.461861 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.461940 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.462090 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"threads_started":1}
I20260929 11:21:19.462752 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.449707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.0.97.61:35157)
0929 11:21:19.451295 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.451304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.451329 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:21:19.451415 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.451423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.451518 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.452513 (+ 995us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.452823 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.454412 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.458663 (+ 4251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.460346 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.462157 (+ 1811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.462260 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.462305 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.462451 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":1238,"threads_started":1}
W20260929 11:21:19.463215 7149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:19.463506 7894 leader_election.cc:288] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079), c52b984fa50548ef81ac294b248bdc16 (127.0.97.60:46339)
W20260929 11:21:19.465539 7343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:19.466320 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.465720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37064)
0929 11:21:19.465889 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.466058 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:21:19.466666 7149 leader_election.cc:334] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:19.467064 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52b984fa50548ef81ac294b248bdc16" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08047313636d4bdf8e6ede057fa5c893" is_pre_election: true
I20260929 11:21:19.467362 7172 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52b984fa50548ef81ac294b248bdc16 in term 1.
I20260929 11:21:19.467769 7343 leader_election.cc:302] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08047313636d4bdf8e6ede057fa5c893, c52b984fa50548ef81ac294b248bdc16; no voters:
I20260929 11:21:19.468129 7895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 11:21:19.468145 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.467545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37076)
0929 11:21:19.467715 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.467924 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 11:21:19.468282 7895 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08ec214716054f739cd17e77b4e5473d)
I20260929 11:21:19.468395 7895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 1 FOLLOWER]: Advancing to term 2
W20260929 11:21:19.468393 7343 leader_election.cc:334] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:19.470703 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.449416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46339 (local address 127.0.0.1:53908)
0929 11:21:19.462916 (+ 13500us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.462934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.463038 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.463861 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.463881 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.463913 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.464240 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.464254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.466251 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.466259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.468616 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.468630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.470216 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.470220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.470327 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.470366 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.470450 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.470537 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13386,"thread_start_us":15094,"threads_started":1}
I20260929 11:21:19.470991 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.449812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.97.60:46339)
0929 11:21:19.463549 (+ 13737us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.463554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.463570 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:21:19.463625 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.463629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.463711 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.464792 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.465062 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.466128 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.468761 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.470072 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.470196 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.470293 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.470523 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.470664 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":13607,"thread_start_us":13480,"threads_started":1}
I20260929 11:21:19.471585 7367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08047313636d4bdf8e6ede057fa5c893" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c52b984fa50548ef81ac294b248bdc16" is_pre_election: true
I20260929 11:21:19.471706 7895 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } }
I20260929 11:21:19.472142 7367 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08047313636d4bdf8e6ede057fa5c893 in current term 2: Already voted for candidate c52b984fa50548ef81ac294b248bdc16 in this term.
I20260929 11:21:19.472368 7895 leader_election.cc:288] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 2 election: Requested vote from peers 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079), 08047313636d4bdf8e6ede057fa5c893 (127.0.97.61:35157)
I20260929 11:21:19.472860 7147 leader_election.cc:302] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08047313636d4bdf8e6ede057fa5c893; no voters: 08ec214716054f739cd17e77b4e5473d, c52b984fa50548ef81ac294b248bdc16
I20260929 11:21:19.473032 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52b984fa50548ef81ac294b248bdc16" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08047313636d4bdf8e6ede057fa5c893"
I20260929 11:21:19.473253 7172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:21:19.476104 7172 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52b984fa50548ef81ac294b248bdc16 in term 2.
I20260929 11:21:19.476295 7894 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:21:19.476435 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.472496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37086)
0929 11:21:19.472707 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.472786 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 11:21:19.476554 7343 leader_election.cc:302] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08047313636d4bdf8e6ede057fa5c893, c52b984fa50548ef81ac294b248bdc16; no voters:
I20260929 11:21:19.476857 7895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:21:19.477469 7895 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [term 2 LEADER]: Becoming Leader. State: Replica: c52b984fa50548ef81ac294b248bdc16, State: Running, Role: LEADER
I20260929 11:21:19.477897 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.477308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37096)
0929 11:21:19.477610 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.477702 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 11:21:19.477898 7895 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } }
W20260929 11:21:19.478099 7343 leader_election.cc:334] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
I20260929 11:21:19.479665 7904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52b984fa50548ef81ac294b248bdc16. Latest consensus state: current_term: 2 leader_uuid: "c52b984fa50548ef81ac294b248bdc16" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:19.480008 7904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:19.480732 7906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:19.484258 7906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:19.485109 7906 catalog_manager.cc:1259] Loaded cluster ID: 4d275977890e45628abadcad783f75db
I20260929 11:21:19.485208 7906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:19.486311 7906 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:19.487110 7906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Loaded TSK: 0
I20260929 11:21:19.487906 7906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:21:19.518357 7357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:19.524927 7357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:19.528728 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.528332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37110)
0929 11:21:19.528469 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.528542 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:21:19.529106 7172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [term 2 FOLLOWER]: Refusing update from remote peer c52b984fa50548ef81ac294b248bdc16: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 11:21:19.529618 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.529310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37124)
0929 11:21:19.529438 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.529499 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:21:19.529914 7904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 11:21:19.530148 7343 consensus_peers.cc:487] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 -> Peer 08ec214716054f739cd17e77b4e5473d (127.0.97.62:33079): Couldn't send request to peer 08ec214716054f739cd17e77b4e5473d. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:21:19.532160 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52b984fa50548ef81ac294b248bdc16. Latest consensus state: current_term: 2 leader_uuid: "c52b984fa50548ef81ac294b248bdc16" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:19.532404 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:19.533967 7904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c52b984fa50548ef81ac294b248bdc16" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:19.534276 7904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:19.534650 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c52b984fa50548ef81ac294b248bdc16" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "08ec214716054f739cd17e77b4e5473d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } peers { permanent_uuid: "08047313636d4bdf8e6ede057fa5c893" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35157 } attrs { promote: false } } peers { permanent_uuid: "c52b984fa50548ef81ac294b248bdc16" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46339 } attrs { promote: false } } }
I20260929 11:21:19.534953 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08047313636d4bdf8e6ede057fa5c893 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:21:19.535459 7908 mvcc.cc:205] Tried to move back new op lower bound from 7334628882543874048 to 7334628882345463808. Current Snapshot: MvccSnapshot[applied={T|T < 7334628882543874048 or (T in {7334628882543874048})}]
I20260929 11:21:19.550132 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.541954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40713 (local address 127.0.0.1:52946)
0929 11:21:19.542096 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.542991 (+ 895us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.543086 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.543431 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.543435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.543459 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.543822 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.543835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.545730 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.545736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.547651 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.547663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.549298 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.549305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.549542 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.549577 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.549676 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.549805 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":21}
I20260929 11:21:19.550192 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.542075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.0.97.2:40713)
0929 11:21:19.542672 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.542681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.543051 (+ 370us) server_negotiation.cc:407] Connection header received
0929 11:21:19.543188 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.543193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.543266 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.544349 (+ 1083us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.544518 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.545596 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.547827 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.549045 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.549554 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.549655 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.549705 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.549832 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
I20260929 11:21:19.551373 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.542352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35101 (local address 127.0.0.1:50204)
0929 11:21:19.542653 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.542670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.542777 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.543584 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.543592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.543628 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.544161 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.544176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.547063 (+ 2887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.547072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.549314 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.549325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.550542 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.550548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.550739 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.550770 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.550855 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.551009 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":224}
I20260929 11:21:19.551285 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 (DEFAULT_TABLE table=test-table [id=0481eb6ecab5454693979b1567f289ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:19.551731 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:19.552119 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.542645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.0.97.1:35101)
0929 11:21:19.543156 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.543165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.543187 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:21:19.543275 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.543282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.543389 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.543739 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.545788 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.546907 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.549465 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.550372 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.550504 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.550604 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.551682 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.551825 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260929 11:21:19.553056 7474 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 (DEFAULT_TABLE table=test-table [id=0481eb6ecab5454693979b1567f289ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:19.553498 7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:19.557003 7917 tablet_bootstrap.cc:492] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Bootstrap starting.
I20260929 11:21:19.557595 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.543845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37223 (local address 127.0.0.1:38224)
0929 11:21:19.544564 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.544610 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.544757 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.549437 (+ 4680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.549442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.549466 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.551298 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.551309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.552823 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.552828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.554966 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.554980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.556751 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.556758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.556965 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.557003 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.557084 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.557213 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":21,"thread_start_us":292,"threads_started":1}
I20260929 11:21:19.559054 7917 tablet_bootstrap.cc:654] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:19.558312 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.543953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.0.97.3:37223)
0929 11:21:19.548968 (+ 5015us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.548975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.548997 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:21:19.549071 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.549078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.549167 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.549259 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.551522 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.552646 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.555136 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.556362 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.556490 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.557811 (+ 1321us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.557866 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.558017 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4861,"thread_start_us":4772,"threads_started":1}
I20260929 11:21:19.559983 7755 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 (DEFAULT_TABLE table=test-table [id=0481eb6ecab5454693979b1567f289ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:19.560458 7755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:19.561409 7917 tablet_bootstrap.cc:492] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: No bootstrap required, opened a new log
I20260929 11:21:19.561547 7917 ts_tablet_manager.cc:1397] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 11:21:19.562444 7917 raft_consensus.cc:348] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.562721 7917 raft_consensus.cc:374] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:19.562812 7917 raft_consensus.cc:729] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf6600aabbe4cfea93c14e272f75a14, State: Initialized, Role: FOLLOWER
I20260929 11:21:19.563069 7917 consensus_queue.cc:260] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.565260 7917 ts_tablet_manager.cc:1428] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 11:21:19.568192 7679 heartbeater.cc:503] Master 127.0.97.60:46339 was elected leader, sending a full tablet report...
I20260929 11:21:19.570031 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.567936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:36319)
0929 11:21:19.569496 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.569668 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1401,"threads_started":1}
I20260929 11:21:19.572160 7918 tablet_bootstrap.cc:492] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Bootstrap starting.
I20260929 11:21:19.572367 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.572010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:44753)
0929 11:21:19.572168 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.572246 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 11:21:19.573115 7921 tablet_bootstrap.cc:492] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Bootstrap starting.
I20260929 11:21:19.574092 7918 tablet_bootstrap.cc:654] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:19.574550 7921 tablet_bootstrap.cc:654] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:19.576099 7918 tablet_bootstrap.cc:492] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: No bootstrap required, opened a new log
I20260929 11:21:19.576260 7918 ts_tablet_manager.cc:1397] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:21:19.576376 7921 tablet_bootstrap.cc:492] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: No bootstrap required, opened a new log
I20260929 11:21:19.576510 7921 ts_tablet_manager.cc:1397] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:21:19.576979 7918 raft_consensus.cc:348] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.577232 7918 raft_consensus.cc:374] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:19.577394 7918 raft_consensus.cc:729] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67946dde31c42c6bc1f9abc3aa03034, State: Initialized, Role: FOLLOWER
I20260929 11:21:19.577337 7921 raft_consensus.cc:348] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.577597 7921 raft_consensus.cc:374] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:19.577690 7921 raft_consensus.cc:729] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4855b9133cb94dc8970dad7192a7e497, State: Initialized, Role: FOLLOWER
I20260929 11:21:19.577656 7918 consensus_queue.cc:260] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.577948 7921 consensus_queue.cc:260] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.578471 7918 ts_tablet_manager.cc:1428] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 11:21:19.578682 7921 ts_tablet_manager.cc:1428] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:21:19.580410 7539 heartbeater.cc:503] Master 127.0.97.60:46339 was elected leader, sending a full tablet report...
I20260929 11:21:19.581691 7820 heartbeater.cc:503] Master 127.0.97.60:46339 was elected leader, sending a full tablet report...
I20260929 11:21:19.584025 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.578953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:34665)
0929 11:21:19.583534 (+ 4581us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.583697 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4391,"thread_start_us":755,"threads_started":1}
I20260929 11:21:19.585139 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.584728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:35537)
0929 11:21:19.584907 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.585007 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:21:19.585644 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.579288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:51361)
0929 11:21:19.585291 (+ 6003us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.585406 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5875,"thread_start_us":5746,"threads_started":1}
I20260929 11:21:19.586637 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.586382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:43093)
0929 11:21:19.586485 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.586546 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 11:21:19.622257 7925 raft_consensus.cc:482] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:19.622509 7925 raft_consensus.cc:504] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.624001 7925 leader_election.cc:288] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713), 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:19.631031 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.624417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51107 (local address 127.0.97.3:37223)
0929 11:21:19.624668 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.624676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.624697 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:21:19.625386 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.625400 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.625481 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.625548 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.626333 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.627038 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.629003 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.630223 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.630375 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.630465 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.630759 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.630844 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:21:19.631187 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.624251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37223 (local address 127.0.97.1:51107)
0929 11:21:19.624568 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.624629 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.625178 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.625726 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.625731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.625769 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.626184 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.626196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.627152 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.627155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.628874 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.628886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.630399 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.630404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.630603 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.630638 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.630734 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.630878 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":32,"spinlock_wait_cycles":2816}
I20260929 11:21:19.631928 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb4fd9b8f4a47eebe8cfda039debcb6" candidate_uuid: "b67946dde31c42c6bc1f9abc3aa03034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4855b9133cb94dc8970dad7192a7e497" is_pre_election: true
I20260929 11:21:19.632175 7775 raft_consensus.cc:2454] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67946dde31c42c6bc1f9abc3aa03034 in term 0.
I20260929 11:21:19.632225 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.624900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34003 (local address 127.0.97.2:40713)
0929 11:21:19.625090 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.625098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.625318 (+ 220us) server_negotiation.cc:407] Connection header received
0929 11:21:19.625513 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.625519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.625602 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.625712 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:19.627186 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.627822 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.629900 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.631201 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.631334 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.631462 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.631835 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.631990 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:21:19.632251 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.624271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40713 (local address 127.0.97.1:34003)
0929 11:21:19.625207 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.625225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:19.625352 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:19.625726 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:19.625731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:19.625762 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:19.626191 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.626198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.627947 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.627952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.629760 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:19.629772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.631560 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.631567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:19.631700 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.631722 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:21:19.631803 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:21:19.631881 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":17,"spinlock_wait_cycles":1280,"thread_start_us":611,"threads_started":1}
I20260929 11:21:19.632764 7428 leader_election.cc:302] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4855b9133cb94dc8970dad7192a7e497, b67946dde31c42c6bc1f9abc3aa03034; no voters:
I20260929 11:21:19.633144 7925 raft_consensus.cc:2781] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:19.633074 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb4fd9b8f4a47eebe8cfda039debcb6" candidate_uuid: "b67946dde31c42c6bc1f9abc3aa03034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf6600aabbe4cfea93c14e272f75a14" is_pre_election: true
I20260929 11:21:19.633248 7925 raft_consensus.cc:482] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:19.633345 7925 raft_consensus.cc:3037] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:19.633471 7634 raft_consensus.cc:2454] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67946dde31c42c6bc1f9abc3aa03034 in term 0.
I20260929 11:21:19.635973 7925 raft_consensus.cc:504] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.636647 7925 leader_election.cc:288] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [CANDIDATE]: Term 1 election: Requested vote from peers 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713), 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223)
I20260929 11:21:19.636950 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb4fd9b8f4a47eebe8cfda039debcb6" candidate_uuid: "b67946dde31c42c6bc1f9abc3aa03034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf6600aabbe4cfea93c14e272f75a14"
I20260929 11:21:19.637156 7634 raft_consensus.cc:3037] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:19.637136 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb4fd9b8f4a47eebe8cfda039debcb6" candidate_uuid: "b67946dde31c42c6bc1f9abc3aa03034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4855b9133cb94dc8970dad7192a7e497"
I20260929 11:21:19.637326 7775 raft_consensus.cc:3037] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:19.640093 7775 raft_consensus.cc:2454] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67946dde31c42c6bc1f9abc3aa03034 in term 1.
I20260929 11:21:19.640093 7634 raft_consensus.cc:2454] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67946dde31c42c6bc1f9abc3aa03034 in term 1.
I20260929 11:21:19.640437 7428 leader_election.cc:302] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4855b9133cb94dc8970dad7192a7e497, b67946dde31c42c6bc1f9abc3aa03034; no voters:
I20260929 11:21:19.640744 7925 raft_consensus.cc:2781] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:19.641144 7925 raft_consensus.cc:686] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 LEADER]: Becoming Leader. State: Replica: b67946dde31c42c6bc1f9abc3aa03034, State: Running, Role: LEADER
I20260929 11:21:19.642004 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.641670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:34457)
0929 11:21:19.641786 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.641889 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 11:21:19.641968 7925 consensus_queue.cc:237] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } }
I20260929 11:21:19.645738 7356 catalog_manager.cc:5495] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 reported cstate change: term changed from 0 to 1, leader changed from <none> to b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b67946dde31c42c6bc1f9abc3aa03034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b67946dde31c42c6bc1f9abc3aa03034" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 } health_report { overall_health: UNKNOWN } } }
I20260929 11:21:19.679837 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.674859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.0.97.1:35101)
0929 11:21:19.675065 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.675071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.675089 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:21:19.675288 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.675292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.675363 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.675518 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:19.675879 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.676551 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.677852 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.678288 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.678428 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.678819 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:19.679319 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:19.679488 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.679531 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.679604 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:21:19.683543 7775 raft_consensus.cc:1261] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Refusing update from remote peer b67946dde31c42c6bc1f9abc3aa03034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:19.683988 7634 raft_consensus.cc:1261] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Refusing update from remote peer b67946dde31c42c6bc1f9abc3aa03034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:19.684258 7930 consensus_queue.cc:1035] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4855b9133cb94dc8970dad7192a7e497" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:19.684813 7925 consensus_queue.cc:1035] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf6600aabbe4cfea93c14e272f75a14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:19.687685 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.686298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:43033)
0929 11:21:19.687279 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.687438 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":825}
I20260929 11:21:19.688977 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.688574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:35645)
0929 11:21:19.688740 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.688814 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 11:21:19.691830 7933 mvcc.cc:205] Tried to move back new op lower bound from 7334628883177639936 to 7334628883017043968. Current Snapshot: MvccSnapshot[applied={T|T < 7334628883177639936}]
I20260929 11:21:19.693961 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.693500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:41523)
0929 11:21:19.693700 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.693786 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 11:21:19.695317 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.694868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:41145)
0929 11:21:19.695104 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.695184 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 11:21:19.698848 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.698447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:40793)
0929 11:21:19.698582 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.698698 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:21:19.700290 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.699911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:60677)
0929 11:21:19.700049 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:19.700134 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 11:21:19.710491 7356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:19.710740 7356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:19.715528 7356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c52b984fa50548ef81ac294b248bdc16: Sending DeleteTablet for 3 replicas of tablet 8bb4fd9b8f4a47eebe8cfda039debcb6
I20260929 11:21:19.716441 7614 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52946
I20260929 11:21:19.716573 7474 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50204
I20260929 11:21:19.716897 7755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38224
I20260929 11:21:19.718359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 7344
I20260929 11:21:19.729001 7943 tablet_replica.cc:330] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: stopping tablet replica
I20260929 11:21:19.729344 7943 raft_consensus.cc:2229] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:19.730221 7943 raft_consensus.cc:2258] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:19.731567 7943 ts_tablet_manager.cc:1905] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:19.731758 7942 tablet_replica.cc:330] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: stopping tablet replica
I20260929 11:21:19.733052 7942 raft_consensus.cc:2229] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:19.734311 7942 raft_consensus.cc:2258] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:19.735976 7942 ts_tablet_manager.cc:1905] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:19.736398 7944 tablet_replica.cc:330] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: stopping tablet replica
I20260929 11:21:19.736660 7944 raft_consensus.cc:2229] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:19.737488 7944 raft_consensus.cc:2258] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:19.738924 7943 ts_tablet_manager.cc:1918] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:19.739084 7943 log.cc:1196] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/8bb4fd9b8f4a47eebe8cfda039debcb6
I20260929 11:21:19.739502 7943 ts_tablet_manager.cc:1939] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P b67946dde31c42c6bc1f9abc3aa03034: Deleting consensus metadata
I20260929 11:21:19.740633 7343 catalog_manager.cc:4841] TS b67946dde31c42c6bc1f9abc3aa03034 (127.0.97.1:35101): tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 (table test-table [id=0481eb6ecab5454693979b1567f289ec]) successfully deleted
I20260929 11:21:19.745229 7942 ts_tablet_manager.cc:1918] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:19.745253 7944 ts_tablet_manager.cc:1905] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:19.745453 7942 log.cc:1196] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/8bb4fd9b8f4a47eebe8cfda039debcb6
I20260929 11:21:19.745941 7942 ts_tablet_manager.cc:1939] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 8cf6600aabbe4cfea93c14e272f75a14: Deleting consensus metadata
I20260929 11:21:19.747085 7340 catalog_manager.cc:4841] TS 8cf6600aabbe4cfea93c14e272f75a14 (127.0.97.2:40713): tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 (table test-table [id=0481eb6ecab5454693979b1567f289ec]) successfully deleted
I20260929 11:21:19.752393 7944 ts_tablet_manager.cc:1918] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:19.752606 7944 log.cc:1196] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/8bb4fd9b8f4a47eebe8cfda039debcb6
I20260929 11:21:19.753085 7944 ts_tablet_manager.cc:1939] T 8bb4fd9b8f4a47eebe8cfda039debcb6 P 4855b9133cb94dc8970dad7192a7e497: Deleting consensus metadata
I20260929 11:21:19.753247 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.719583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.97.1:35101)
0929 11:21:19.719769 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.719777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.732473 (+ 12696us) server_negotiation.cc:407] Connection header received
0929 11:21:19.732751 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.732756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.732843 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.732954 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:19.737274 (+ 4320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.738440 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.742520 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.743124 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.743273 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.746203 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:19.746224 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:19.746239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:19.746268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:19.750070 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:19.750880 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:19.750890 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:19.750893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:19.751081 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:19.751499 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:19.751504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:19.751508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:19.751943 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:19.752240 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.752626 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.752858 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:21:19.753958 7534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:19.754311 7341 catalog_manager.cc:4841] TS 4855b9133cb94dc8970dad7192a7e497 (127.0.97.3:37223): tablet 8bb4fd9b8f4a47eebe8cfda039debcb6 (table test-table [id=0481eb6ecab5454693979b1567f289ec]) successfully deleted
I20260929 11:21:19.855053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 7344
I20260929 11:21:19.872916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 7551
I20260929 11:21:19.886224 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:19.873626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.0.97.2:40713)
0929 11:21:19.873843 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:19.873849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:19.873919 (+ 70us) server_negotiation.cc:407] Connection header received
0929 11:21:19.874131 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:19.874136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:19.874212 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:19.874328 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:19.875552 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.876235 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.877666 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:19.878077 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:19.878208 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:19.880580 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:19.880619 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:19.880640 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:19.880684 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:19.883305 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:19.884015 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:19.884025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:19.884029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:19.884161 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:19.884529 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:19.884535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:19.884538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:19.885096 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:19.885419 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:19.885666 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 11:21:19.885887 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:21:19.886948 7674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:20.004117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 7551
I20260929 11:21:20.009262 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:20.008871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37126)
0929 11:21:20.009040 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:20.009128 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:21:20.010264 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:20.009905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.0.1:37142)
0929 11:21:20.010047 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:20.010143 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 11:21:20.026288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 7692
I20260929 11:21:20.038674 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:20.027018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.97.3:37223)
0929 11:21:20.027192 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:20.027196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:20.027230 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:21:20.027401 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:20.027405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:20.027462 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:20.027545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:20.028911 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:20.029618 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:20.030724 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:20.031091 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:20.031174 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:20.033550 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:20.033564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:20.033581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:20.033621 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:20.036453 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:20.037068 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:20.037075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:20.037077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:20.037215 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:20.037524 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:20.037527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:20.037529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:20.037847 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:20.038062 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:20.038298 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 11:21:20.038440 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:21:20.039340 7815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:20.130414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 7692
I20260929 11:21:20.147696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 7019
I20260929 11:21:20.148823 7192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:20.261341 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 7019
I20260929 11:21:20.281137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 7118
I20260929 11:21:20.282191 7387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:20.399847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 7118
2026-09-29T11:21:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9243 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:20 dist-test-slave-srst krb5kdc[7953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:20 dist-test-slave-srst krb5kdc[7953](info): set up 2 sockets
Sep 29 11:21:20 dist-test-slave-srst krb5kdc[7953](info): commencing operation
krb5kdc: starting...
W20260929 11:21:21.493746 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:21 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 11:21:21.695940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41243
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:22.003472 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:22.003850 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:22.004068 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:22.012929 7966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:22.013028 7966 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:21:22.013068 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:22.013126 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:22.013173 7966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:22.013236 7966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:22.022920 7966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41243
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:22.024129 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:22.025462 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:22.036996 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:22.036949 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:22.037041 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:22.037963 7966 server_base.cc:1034] running on GCE node
I20260929 11:21:22.038655 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:22.039942 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:22.041148 7966 hybrid_clock.cc:648] HybridClock initialized: now 1790680882041096 us; error 29 us; skew 500 ppm
I20260929 11:21:22.041421 7966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:22 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:22.048923 7966 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 11:21:22.050359 7966 webserver.cc:460] Webserver started at http://127.0.97.62:34581/ using document root <none> and password file <none>
I20260929 11:21:22.050997 7966 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:22.051079 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:22.051347 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:22.054219 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "f7cf2bac47874eefbf0f2c19667c44ac"
format_stamp: "Formatted at 2026-09-29 11:21:22 on dist-test-slave-srst"
server_key: "8a87f69ba736111c36a7e7d722b5dd8e"
server_key_iv: "c3e6255b1136302b4ced6d955eb304ee"
server_key_version: "encryptionkey@0"
I20260929 11:21:22.054934 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "f7cf2bac47874eefbf0f2c19667c44ac"
format_stamp: "Formatted at 2026-09-29 11:21:22 on dist-test-slave-srst"
server_key: "8a87f69ba736111c36a7e7d722b5dd8e"
server_key_iv: "c3e6255b1136302b4ced6d955eb304ee"
server_key_version: "encryptionkey@0"
I20260929 11:21:22.059725 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:21:22.062924 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:22.064185 7966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:21:22.064307 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "f7cf2bac47874eefbf0f2c19667c44ac"
format_stamp: "Formatted at 2026-09-29 11:21:22 on dist-test-slave-srst"
server_key: "8a87f69ba736111c36a7e7d722b5dd8e"
server_key_iv: "c3e6255b1136302b4ced6d955eb304ee"
server_key_version: "encryptionkey@0"
I20260929 11:21:22.064462 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:22.085453 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:22.089542 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:22.089788 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:22.114277 7966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41243
I20260929 11:21:22.114298 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41243 every 8 connection(s)
I20260929 11:21:22.115571 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:22.119102 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:22.124507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 7966
I20260929 11:21:22.124742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:22.125571 388 external_mini_cluster.cc:1428] Setting key a0addcb18d1c3b361c8dcdfd089ff7a4
I20260929 11:21:22.130371 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Bootstrap starting.
I20260929 11:21:22.133394 8043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:22.134387 8043 log.cc:824] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:22.137504 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: No bootstrap required, opened a new log
Sep 29 11:21:22 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:22.147697 8043 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41243 } }
I20260929 11:21:22.148155 8043 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:22.148272 8043 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cf2bac47874eefbf0f2c19667c44ac, State: Initialized, Role: FOLLOWER
I20260929 11:21:22.149060 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:22.127781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.97.62:41243)
0929 11:21:22.128564 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:22.128575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:22.128679 (+ 104us) server_negotiation.cc:407] Connection header received
0929 11:21:22.130278 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:22.130299 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:22.130736 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:22.131101 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:22.132492 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.133902 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.135386 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.136308 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.136501 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:22.140430 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:22.140441 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:22.140465 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:22.140514 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:22.144428 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:22.145140 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:22.145149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:22.145159 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:22.145392 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:22.145717 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:22.145723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:22.145726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:22.146507 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:22.146798 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:22.147161 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 11:21:22.147614 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":363,"threads_started":1}
I20260929 11:21:22.148974 8043 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41243 } }
I20260929 11:21:22.149271 8043 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:22.149456 8043 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:22.149652 8043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:22.152242 8043 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41243 } }
I20260929 11:21:22.152910 8043 leader_election.cc:302] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cf2bac47874eefbf0f2c19667c44ac; no voters:
I20260929 11:21:22.153491 8043 leader_election.cc:288] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:22.153692 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:22.160898 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [term 1 LEADER]: Becoming Leader. State: Replica: f7cf2bac47874eefbf0f2c19667c44ac, State: Running, Role: LEADER
I20260929 11:21:22.161603 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41243 } }
I20260929 11:21:22.164058 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cf2bac47874eefbf0f2c19667c44ac. Latest consensus state: current_term: 1 leader_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41243 } } }
I20260929 11:21:22.164460 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:22.165030 8043 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:22.165053 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cf2bac47874eefbf0f2c19667c44ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41243 } } }
I20260929 11:21:22.165534 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:22.173547 8052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:22.178930 8052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:22.188102 8052 catalog_manager.cc:1347] Generated new cluster ID: 6d35205bf63f419d9acece207376c8be
I20260929 11:21:22.188265 8052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:22.223649 8052 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:22.225076 8052 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:22.230888 8052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Generated new TSK 0
I20260929 11:21:22.231686 8052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 11:21:22.288398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41243
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:22.610374 8069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:22.610633 8069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:22.610780 8069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:22.617769 8069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:22.617862 8069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:22.618033 8069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:21:22.618167 8069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:22.627676 8069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:22.628953 8069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:22.630278 8069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:22.642503 8075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:22.642470 8074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:22.642652 8077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:22.643322 8069 server_base.cc:1034] running on GCE node
I20260929 11:21:22.643935 8069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:22.644845 8069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:22.646044 8069 hybrid_clock.cc:648] HybridClock initialized: now 1790680882645978 us; error 54 us; skew 500 ppm
I20260929 11:21:22.646350 8069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:22 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:22.649551 8069 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 11:21:22.651117 8069 webserver.cc:460] Webserver started at http://127.0.97.1:37627/ using document root <none> and password file <none>
I20260929 11:21:22.651713 8069 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:22.651803 8069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:22.652050 8069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:22.654912 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "403e3f833f7e4fb28ccdc432393b9d90"
format_stamp: "Formatted at 2026-09-29 11:21:22 on dist-test-slave-srst"
server_key: "7e605e905302e1151f4ae70419f0712f"
server_key_iv: "e46cef2336b17d1049d010d5efa1cb78"
server_key_version: "encryptionkey@0"
I20260929 11:21:22.655622 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "403e3f833f7e4fb28ccdc432393b9d90"
format_stamp: "Formatted at 2026-09-29 11:21:22 on dist-test-slave-srst"
server_key: "7e605e905302e1151f4ae70419f0712f"
server_key_iv: "e46cef2336b17d1049d010d5efa1cb78"
server_key_version: "encryptionkey@0"
I20260929 11:21:22.661002 8069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:21:22.664361 8084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:22.665786 8069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:22.665951 8069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "403e3f833f7e4fb28ccdc432393b9d90"
format_stamp: "Formatted at 2026-09-29 11:21:22 on dist-test-slave-srst"
server_key: "7e605e905302e1151f4ae70419f0712f"
server_key_iv: "e46cef2336b17d1049d010d5efa1cb78"
server_key_version: "encryptionkey@0"
I20260929 11:21:22.666216 8069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:22.685793 8069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:22.689766 8069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:22.690037 8069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:22.692535 8069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:22.692658 8069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 11:21:22.692818 8069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:22.692943 8069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:21:22 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:22.717376 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:22.693282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54049 (local address 127.0.97.62:41243)
0929 11:21:22.693788 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:22.693794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:22.694998 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 11:21:22.698230 (+ 3232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:22.698238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:22.698330 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:22.698470 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:22.700681 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.701985 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.704262 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.705069 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.705224 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:22.710312 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:22.710319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:22.710323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:22.710366 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:22.713236 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:22.714257 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:22.714263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:22.714266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:22.714416 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:22.714963 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:22.714968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:22.714971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:22.715358 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:22.715501 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:22.716522 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0929 11:21:22.716820 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
I20260929 11:21:22.718310 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:22.693908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.97.1:54049)
0929 11:21:22.694804 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:22.694868 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:22.697823 (+ 2955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:22.698765 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:22.698779 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:22.699600 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:22.700435 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:22.700473 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.702199 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.702206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:22.704087 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:22.704109 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.705338 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.705349 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:22.705651 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:22.706518 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:22.706538 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:22.710000 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:22.713385 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:22.713394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:22.713410 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:22.714129 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:22.714522 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:22.714529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:22.714547 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:22.714853 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:22.715485 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:22.715497 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:22.716060 (+ 563us) client_negotiation.cc:769] Sending connection context
0929 11:21:22.716381 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 11:21:22.717014 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":364,"threads_started":1}
I20260929 11:21:22.738885 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:22.723854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46389 (local address 127.0.97.62:41243)
0929 11:21:22.724059 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:22.724066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:22.724766 (+ 700us) server_negotiation.cc:407] Connection header received
0929 11:21:22.724958 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:22.724961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:22.725018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:22.725095 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:22.726515 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.727209 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.729757 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.730138 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.730267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:22.733131 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:22.733136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:22.733139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:22.733167 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:22.735605 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:22.736558 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:22.736561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:22.736563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:22.736734 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:22.737205 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:22.737214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:22.737217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:22.737631 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:22.737799 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:22.738312 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 11:21:22.738532 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:21:22.739013 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:22.723850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.97.1:46389)
0929 11:21:22.724649 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:22.724675 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:22.724839 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:22.725159 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:22.725167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:22.725883 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:22.726356 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:22.726372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.727352 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.727359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:22.729594 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:22.729614 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:22.730355 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:22.730364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:22.730629 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:22.731434 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:22.731465 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:22.732936 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:22.735738 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:22.735744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:22.735751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:22.736433 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:22.736856 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:22.736861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:22.736865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:22.737057 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:22.737806 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:22.737815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:22.738044 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 11:21:22.738194 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 11:21:22.738537 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":406,"threads_started":1}
I20260929 11:21:22.747913 8069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34141
I20260929 11:21:22.747936 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34141 every 8 connection(s)
I20260929 11:21:22.749198 8069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:22.751137 8205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41243
I20260929 11:21:22.751367 8205 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:22.752009 8205 heartbeater.cc:511] Master 127.0.97.62:41243 requested a full tablet report, sending...
I20260929 11:21:22.753975 7998 ts_manager.cc:194] Registered new tserver with Master: 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141)
I20260929 11:21:22.755342 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54049
I20260929 11:21:22.758191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 8069
I20260929 11:21:22.758347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:22.758906 388 external_mini_cluster.cc:1428] Setting key 544a74ba7928cb3f3560cd2e33da5b05
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 11:21:22.803375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41243
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:23.131105 8209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:23.131402 8209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:23.131564 8209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:23.138716 8209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:23.138892 8209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:23.139029 8209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:21:23.139158 8209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:23.148566 8209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:23.149864 8209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:23.151180 8209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:23.163676 8214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:23.163697 8217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:23.164278 8215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:23.165884 8209 server_base.cc:1034] running on GCE node
I20260929 11:21:23.166534 8209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:23.167433 8209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:23.168697 8209 hybrid_clock.cc:648] HybridClock initialized: now 1790680883168622 us; error 38 us; skew 500 ppm
I20260929 11:21:23.169003 8209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:23.172166 8209 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 11:21:23.173763 8209 webserver.cc:460] Webserver started at http://127.0.97.2:35477/ using document root <none> and password file <none>
I20260929 11:21:23.174512 8209 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:23.174623 8209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:23.174986 8209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:23.177866 8209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "691fe906ef774e7b87a73ff24ec3e58b"
format_stamp: "Formatted at 2026-09-29 11:21:23 on dist-test-slave-srst"
server_key: "29c13141a672f90b028cb02453c9af13"
server_key_iv: "e271a1a43fac86c455c7384352437f90"
server_key_version: "encryptionkey@0"
I20260929 11:21:23.178637 8209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "691fe906ef774e7b87a73ff24ec3e58b"
format_stamp: "Formatted at 2026-09-29 11:21:23 on dist-test-slave-srst"
server_key: "29c13141a672f90b028cb02453c9af13"
server_key_iv: "e271a1a43fac86c455c7384352437f90"
server_key_version: "encryptionkey@0"
I20260929 11:21:23.183806 8209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:21:23.187084 8224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:23.188334 8209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:21:23.188495 8209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "691fe906ef774e7b87a73ff24ec3e58b"
format_stamp: "Formatted at 2026-09-29 11:21:23 on dist-test-slave-srst"
server_key: "29c13141a672f90b028cb02453c9af13"
server_key_iv: "e271a1a43fac86c455c7384352437f90"
server_key_version: "encryptionkey@0"
I20260929 11:21:23.188789 8209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:23.203747 8209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:23.207999 8209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:23.208276 8209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:23.210796 8209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:23.210902 8209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:23.211043 8209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:23.211120 8209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:23.233484 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.211522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50283 (local address 127.0.97.62:41243)
0929 11:21:23.211717 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.211723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.213191 (+ 1468us) server_negotiation.cc:407] Connection header received
0929 11:21:23.215348 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.215354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.215437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.215537 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:23.217951 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.218682 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.221171 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.221753 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.221889 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.226801 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:23.226806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:23.226809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:23.226846 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:23.230162 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.231164 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.231169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.231172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.231295 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.231733 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.231738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.231741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.232017 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:23.232194 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.232876 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.233115 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:21:23.234566 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.212098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.97.2:50283)
0929 11:21:23.212999 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.213064 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.214990 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.216049 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.216057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.216885 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:23.217774 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.217792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.218837 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.218845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.221010 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.221026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.222202 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.222207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.222406 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.223149 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:23.223186 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:23.226518 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:23.230357 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.230371 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.230390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.230973 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.231402 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.231407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.231411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.231591 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.232174 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:23.232182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:23.232487 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.232787 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.233325 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":370,"threads_started":1}
I20260929 11:21:23.258164 8209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44865
I20260929 11:21:23.258193 8343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44865 every 8 connection(s)
I20260929 11:21:23.259086 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.244393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.97.2:58953)
0929 11:21:23.245286 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.245306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.245536 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.245897 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.245904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.246745 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:23.247295 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.247307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.248517 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.248521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.250368 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.250380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.251088 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.251094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.251258 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.251957 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:23.251972 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:23.252792 (+ 820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:23.256402 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.256412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.256416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.256859 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.257217 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.257222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.257226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.257414 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.258194 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:23.258200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:23.258421 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.258541 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.258763 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":491,"threads_started":1}
I20260929 11:21:23.259222 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.244295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58953 (local address 127.0.97.62:41243)
0929 11:21:23.244448 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.244451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.245416 (+ 965us) server_negotiation.cc:407] Connection header received
0929 11:21:23.245674 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.245679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.245755 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.245855 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:23.247434 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.248232 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.250519 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.250913 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.251044 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.252984 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:23.252990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:23.252994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:23.253039 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:23.256275 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.256977 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.256982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.256985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.257102 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.257560 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.257568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.257574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.258060 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:23.258217 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.258681 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.258922 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:21:23.259810 8209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:23.261929 8344 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41243
I20260929 11:21:23.262162 8344 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:23.262899 8344 heartbeater.cc:511] Master 127.0.97.62:41243 requested a full tablet report, sending...
I20260929 11:21:23.264158 7998 ts_manager.cc:194] Registered new tserver with Master: 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865)
I20260929 11:21:23.264487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 8209
I20260929 11:21:23.264708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:23.265313 388 external_mini_cluster.cc:1428] Setting key 03eb1b6b8c58d32128a69a0e79e38539
I20260929 11:21:23.265301 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50283
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 11:21:23.309902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41243
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:23.630120 8348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:23.630463 8348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:23.630642 8348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:23.638002 8348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:23.638154 8348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:23.638298 8348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:21:23.638437 8348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:23.648144 8348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:23.649590 8348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:23.650900 8348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:23.663833 8353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:23.663802 8356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:23.663872 8354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:23.664983 8348 server_base.cc:1034] running on GCE node
I20260929 11:21:23.665591 8348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:23.666574 8348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:23.667861 8348 hybrid_clock.cc:648] HybridClock initialized: now 1790680883667742 us; error 48 us; skew 500 ppm
I20260929 11:21:23.668206 8348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:23.671801 8348 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 11:21:23.673381 8348 webserver.cc:460] Webserver started at http://127.0.97.3:41521/ using document root <none> and password file <none>
I20260929 11:21:23.674188 8348 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:23.674322 8348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:23.674643 8348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:23.678054 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "a36610f683fe44548b1c09af2c7e34fa"
format_stamp: "Formatted at 2026-09-29 11:21:23 on dist-test-slave-srst"
server_key: "7e9c7e14061be75116f918cf7211c6a2"
server_key_iv: "bff3db73f67641fa3bb705ef9d9010fa"
server_key_version: "encryptionkey@0"
I20260929 11:21:23.678875 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a36610f683fe44548b1c09af2c7e34fa"
format_stamp: "Formatted at 2026-09-29 11:21:23 on dist-test-slave-srst"
server_key: "7e9c7e14061be75116f918cf7211c6a2"
server_key_iv: "bff3db73f67641fa3bb705ef9d9010fa"
server_key_version: "encryptionkey@0"
I20260929 11:21:23.683928 8348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:21:23.687619 8363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:23.689035 8348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:23.689215 8348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "a36610f683fe44548b1c09af2c7e34fa"
format_stamp: "Formatted at 2026-09-29 11:21:23 on dist-test-slave-srst"
server_key: "7e9c7e14061be75116f918cf7211c6a2"
server_key_iv: "bff3db73f67641fa3bb705ef9d9010fa"
server_key_version: "encryptionkey@0"
I20260929 11:21:23.689450 8348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:23.714717 8348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:23.718801 8348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:23.719074 8348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:23.722010 8348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:23.722110 8348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:23.722306 8348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:23.722426 8348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:23.749441 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.723124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.97.3:47129)
0929 11:21:23.723838 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.723873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.725705 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.726494 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.726503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.727158 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:23.728114 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.728140 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.729650 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.729660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.732610 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.732625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.733601 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.733611 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.733969 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.734776 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:23.734818 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:23.738558 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:23.743092 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.743103 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.743126 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.743800 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.744286 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.744295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.744300 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.744717 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.745689 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:23.745700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:23.746292 (+ 592us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.746679 (+ 387us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.747397 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":294,"threads_started":1}
I20260929 11:21:23.751168 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.722482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47129 (local address 127.0.97.62:41243)
0929 11:21:23.722674 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.722681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.723992 (+ 1311us) server_negotiation.cc:407] Connection header received
0929 11:21:23.726070 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.726078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.726180 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.726300 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:23.728355 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.729431 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.732759 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.733357 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.733509 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.738844 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:23.738849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:23.738852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:23.738885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:23.742940 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.743935 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.743941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.743945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.744146 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.744908 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.744916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.744922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.745436 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:23.745599 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.750341 (+ 4742us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.750655 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 11:21:23.757920 8205 heartbeater.cc:503] Master 127.0.97.62:41243 was elected leader, sending a full tablet report...
I20260929 11:21:23.775635 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.758077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.97.3:36819)
0929 11:21:23.761314 (+ 3237us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.761337 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.761478 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.761859 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.761865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.762480 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:23.762992 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.763003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.764318 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.764325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.766149 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.766169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.767293 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.767298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.767419 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.768267 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:23.768285 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:23.769526 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:23.772741 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.772745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.772749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.773350 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.773786 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:23.773790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:23.773793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:23.773927 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:23.774551 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:23.774556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:23.774734 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.774858 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.775167 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3093,"thread_start_us":2942,"threads_started":1}
I20260929 11:21:23.776469 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.758602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36819 (local address 127.0.97.62:41243)
0929 11:21:23.758809 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.758814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.761420 (+ 2606us) server_negotiation.cc:407] Connection header received
0929 11:21:23.761609 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.761617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.761706 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.761811 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:23.763491 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.764189 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.766328 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.766922 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.766996 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.769720 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:23.769725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:23.769728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:23.769756 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:23.772489 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.773511 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.773516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.773519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.773666 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.774040 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.774044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.774048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.774331 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:23.774478 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.775958 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.776162 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:21:23.780177 8348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41259
I20260929 11:21:23.780256 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41259 every 8 connection(s)
I20260929 11:21:23.781597 8348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:23.783869 8483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41243
I20260929 11:21:23.784096 8483 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:23.784786 8483 heartbeater.cc:511] Master 127.0.97.62:41243 requested a full tablet report, sending...
I20260929 11:21:23.785988 7998 ts_manager.cc:194] Registered new tserver with Master: a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:23.786763 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47129
I20260929 11:21:23.790350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 8348
I20260929 11:21:23.790513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:23.790967 388 external_mini_cluster.cc:1428] Setting key 54b6543e2c31cd7b3cd332e5583bec88
I20260929 11:21:23.793762 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:21:23.840517 8093 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:23.830218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.0.97.62:41243)
0929 11:21:23.830446 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.830451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.830639 (+ 188us) server_negotiation.cc:407] Connection header received
0929 11:21:23.830857 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.830865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.830930 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.831033 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:23.832275 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.833110 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.834348 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.834838 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.834991 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.840170 (+ 5179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40762: BlockingRecv error: recv got EOF from 127.0.0.1:40762 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 11:21:23 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:23.857566 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.843780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.97.62:41243)
0929 11:21:23.844025 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.844033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.844308 (+ 275us) server_negotiation.cc:407] Connection header received
0929 11:21:23.844534 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.844540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.844683 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.844795 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:23.845914 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.846625 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.847891 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.848287 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.848433 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.852272 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:23.852277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:23.852280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:23.852314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:23.854956 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.855592 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.855597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.855600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.855721 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:23.856095 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:23.856100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:23.856103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:23.856391 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:23.856701 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.857006 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.857236 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:21:23.861320 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:23.868381 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:23.905231 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.881667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.97.1:34141)
0929 11:21:23.890595 (+ 8928us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.890608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.895993 (+ 5385us) server_negotiation.cc:407] Connection header received
0929 11:21:23.896802 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.896810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.897077 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.897261 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:23.898197 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.899895 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.901984 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.903442 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.903572 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.903820 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.904787 (+ 967us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.904944 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8715,"thread_start_us":8559,"threads_started":1}
I20260929 11:21:23.905258 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.880198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.0.1:35342)
0929 11:21:23.881173 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.881208 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.881374 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.897134 (+ 15760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.897142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.897176 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:23.897752 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.897764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.899819 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.899824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.902348 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.902361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.904077 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.904083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.904464 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.904532 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.904709 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.904872 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":213,"thread_start_us":535,"threads_started":1}
I20260929 11:21:23.905225 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.880673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.97.3:41259)
0929 11:21:23.896067 (+ 15394us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.896079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.896119 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:21:23.896257 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.896268 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.896640 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.896849 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:23.897982 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.899655 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.902518 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.903855 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.904187 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.904447 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.904763 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.904906 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15157,"thread_start_us":16467,"threads_started":1}
I20260929 11:21:23.905660 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.881911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.97.2:44865)
0929 11:21:23.896425 (+ 14514us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:23.896432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:23.896628 (+ 196us) server_negotiation.cc:407] Connection header received
0929 11:21:23.897145 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:23.897155 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:23.897389 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:23.897632 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:23.898492 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.899566 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.902619 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.903811 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.903931 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.904138 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:23.904969 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 11:21:23.905117 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14369,"thread_start_us":14243,"threads_started":1}
I20260929 11:21:23.905992 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.881533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44865 (local address 127.0.0.1:40648)
0929 11:21:23.896486 (+ 14953us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.896504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.896909 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.897866 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.897872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.897900 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:23.898290 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.898299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.900288 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.900293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.902483 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.902496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.904042 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.904049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.904345 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.904489 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.904647 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.904869 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14837,"mutex_wait_us":123,"thread_start_us":14858,"threads_started":1}
I20260929 11:21:23.906243 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:23.881019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34141 (local address 127.0.0.1:54476)
0929 11:21:23.895801 (+ 14782us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:23.895846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:23.896537 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:23.897389 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:23.897417 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:23.897448 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:23.898018 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.898034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.900025 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.900029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.901830 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:23.901841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:23.903973 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:23.903979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:23.904415 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:23.904529 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 11:21:23.904647 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 11:21:23.905934 (+ 1287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14581,"mutex_wait_us":1312,"thread_start_us":15561,"threads_started":1}
I20260929 11:21:23.908999 8413 tablet_service.cc:1467] Processing CreateTablet for tablet e75bc2fdf31741c4ba745872c1ad13f5 (DEFAULT_TABLE table=test-table [id=907a2a9feaf2480095642753bfec7ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:23.910256 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75bc2fdf31741c4ba745872c1ad13f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:23.910818 8274 tablet_service.cc:1467] Processing CreateTablet for tablet e75bc2fdf31741c4ba745872c1ad13f5 (DEFAULT_TABLE table=test-table [id=907a2a9feaf2480095642753bfec7ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:23.912058 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75bc2fdf31741c4ba745872c1ad13f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:23.911866 8135 tablet_service.cc:1467] Processing CreateTablet for tablet e75bc2fdf31741c4ba745872c1ad13f5 (DEFAULT_TABLE table=test-table [id=907a2a9feaf2480095642753bfec7ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:23.913093 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75bc2fdf31741c4ba745872c1ad13f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:23.921482 8507 tablet_bootstrap.cc:492] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Bootstrap starting.
I20260929 11:21:23.924362 8507 tablet_bootstrap.cc:654] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:23.925834 8507 log.cc:824] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:23.926656 8509 tablet_bootstrap.cc:492] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Bootstrap starting.
I20260929 11:21:23.928193 8508 tablet_bootstrap.cc:492] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Bootstrap starting.
I20260929 11:21:23.930377 8509 tablet_bootstrap.cc:654] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:23.930975 8508 tablet_bootstrap.cc:654] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:23.931366 8507 tablet_bootstrap.cc:492] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: No bootstrap required, opened a new log
I20260929 11:21:23.931543 8509 log.cc:824] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:23.931615 8507 ts_tablet_manager.cc:1397] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 11:21:23.931988 8508 log.cc:824] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:23.934429 8509 tablet_bootstrap.cc:492] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: No bootstrap required, opened a new log
I20260929 11:21:23.934659 8509 ts_tablet_manager.cc:1397] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 11:21:23.934964 8508 tablet_bootstrap.cc:492] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: No bootstrap required, opened a new log
I20260929 11:21:23.935179 8508 ts_tablet_manager.cc:1397] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 11:21:23.941618 8509 raft_consensus.cc:348] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:23.942063 8509 raft_consensus.cc:374] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:23.942159 8509 raft_consensus.cc:729] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Initialized, Role: FOLLOWER
I20260929 11:21:23.942369 8507 raft_consensus.cc:348] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:23.942740 8507 raft_consensus.cc:374] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:23.942842 8507 raft_consensus.cc:729] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Initialized, Role: FOLLOWER
I20260929 11:21:23.942842 8509 consensus_queue.cc:260] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:23.943413 8507 consensus_queue.cc:260] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:23.945353 8483 heartbeater.cc:503] Master 127.0.97.62:41243 was elected leader, sending a full tablet report...
I20260929 11:21:23.946219 8507 ts_tablet_manager.cc:1428] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 11:21:23.945436 8508 raft_consensus.cc:348] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:23.946727 8508 raft_consensus.cc:374] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:23.946822 8508 raft_consensus.cc:729] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691fe906ef774e7b87a73ff24ec3e58b, State: Initialized, Role: FOLLOWER
I20260929 11:21:23.947436 8509 ts_tablet_manager.cc:1428] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent starting tablet: real 0.013s user 0.003s sys 0.009s
I20260929 11:21:23.947469 8508 consensus_queue.cc:260] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:23.948509 8344 heartbeater.cc:503] Master 127.0.97.62:41243 was elected leader, sending a full tablet report...
I20260929 11:21:23.949088 8508 ts_tablet_manager.cc:1428] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260929 11:21:24.001616 8206 tablet.cc:2369] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:21:24.011946 8345 tablet.cc:2369] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:21:24.033419 8484 tablet.cc:2369] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:24.282135 8514 raft_consensus.cc:482] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:24.282366 8514 raft_consensus.cc:504] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:24.283908 8514 leader_election.cc:288] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865), 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141)
I20260929 11:21:24.289343 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.284226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54861 (local address 127.0.97.2:44865)
0929 11:21:24.284454 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.284460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.284692 (+ 232us) server_negotiation.cc:407] Connection header received
0929 11:21:24.284885 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.284889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.284940 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.285033 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:24.285436 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.286122 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.287697 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.288448 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.288597 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.288729 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.288972 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.289092 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 11:21:24.289499 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.284093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44865 (local address 127.0.97.3:54861)
0929 11:21:24.284563 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.284580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.284743 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.285048 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.285052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.285126 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:24.285288 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.285296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.286241 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.286249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.287536 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.287547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.288634 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.288640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.288816 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.288839 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.288944 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.289081 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":2,"spinlock_wait_cycles":19712,"thread_start_us":389,"threads_started":1}
I20260929 11:21:24.289808 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.284372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49567 (local address 127.0.97.1:34141)
0929 11:21:24.284570 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.284578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.285043 (+ 465us) server_negotiation.cc:407] Connection header received
0929 11:21:24.285220 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.285224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.285283 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.285372 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:24.285810 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.286458 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.288111 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.288915 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.289059 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.289174 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.289451 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.289585 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:21:24.289887 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.284149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34141 (local address 127.0.97.3:49567)
0929 11:21:24.284942 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.284954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.285085 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.285391 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.285412 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.285434 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:24.285673 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.285682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.286586 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.286597 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.287956 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.287967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.289132 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.289149 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.289300 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.289321 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.289434 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.289579 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":565,"threads_started":1}
I20260929 11:21:24.290530 8296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75bc2fdf31741c4ba745872c1ad13f5" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b" is_pre_election: true
I20260929 11:21:24.290781 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75bc2fdf31741c4ba745872c1ad13f5" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403e3f833f7e4fb28ccdc432393b9d90" is_pre_election: true
I20260929 11:21:24.290980 8296 raft_consensus.cc:2454] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 0.
I20260929 11:21:24.291265 8156 raft_consensus.cc:2454] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 0.
I20260929 11:21:24.291623 8364 leader_election.cc:302] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 691fe906ef774e7b87a73ff24ec3e58b, a36610f683fe44548b1c09af2c7e34fa; no voters:
I20260929 11:21:24.292021 8514 raft_consensus.cc:2781] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:24.292168 8514 raft_consensus.cc:482] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:24.292285 8514 raft_consensus.cc:3037] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.294580 8514 raft_consensus.cc:504] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:24.295086 8514 leader_election.cc:288] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 election: Requested vote from peers 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865), 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141)
I20260929 11:21:24.295539 8296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75bc2fdf31741c4ba745872c1ad13f5" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b"
I20260929 11:21:24.295542 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75bc2fdf31741c4ba745872c1ad13f5" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403e3f833f7e4fb28ccdc432393b9d90"
I20260929 11:21:24.295734 8156 raft_consensus.cc:3037] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.295734 8296 raft_consensus.cc:3037] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.298885 8296 raft_consensus.cc:2454] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 1.
I20260929 11:21:24.298897 8156 raft_consensus.cc:2454] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 1.
I20260929 11:21:24.299300 8366 leader_election.cc:302] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, a36610f683fe44548b1c09af2c7e34fa; no voters:
I20260929 11:21:24.299575 8514 raft_consensus.cc:2781] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:24.299940 8514 raft_consensus.cc:686] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 1 LEADER]: Becoming Leader. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Running, Role: LEADER
I20260929 11:21:24.300359 8514 consensus_queue.cc:237] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } }
I20260929 11:21:24.304360 7998 catalog_manager.cc:5495] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa reported cstate change: term changed from 0 to 1, leader changed from <none> to a36610f683fe44548b1c09af2c7e34fa (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } health_report { overall_health: UNKNOWN } } }
I20260929 11:21:24.358508 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.353510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.3:41259)
0929 11:21:24.353685 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.353689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.353705 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:21:24.353786 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.353790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.353905 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.354033 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:24.354493 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.355167 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.356436 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.356883 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.356970 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.357363 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.357887 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.358019 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.358159 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.358217 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:21:24.366266 8156 raft_consensus.cc:1261] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Refusing update from remote peer a36610f683fe44548b1c09af2c7e34fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:24.366266 8296 raft_consensus.cc:1261] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Refusing update from remote peer a36610f683fe44548b1c09af2c7e34fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:24.367149 8518 consensus_queue.cc:1035] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:24.367470 8514 consensus_queue.cc:1035] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:24.378222 8522 mvcc.cc:205] Tried to move back new op lower bound from 7334628902354505728 to 7334628902097793024. Current Snapshot: MvccSnapshot[applied={T|T < 7334628902354505728}]
I20260929 11:21:24.378491 8521 mvcc.cc:205] Tried to move back new op lower bound from 7334628902354505728 to 7334628902097793024. Current Snapshot: MvccSnapshot[applied={T|T < 7334628902354505728}]
I20260929 11:21:24.381497 8523 mvcc.cc:205] Tried to move back new op lower bound from 7334628902354505728 to 7334628902097793024. Current Snapshot: MvccSnapshot[applied={T|T < 7334628902354505728}]
I20260929 11:21:24.390928 7997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:24.391242 7997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:24.394896 7997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Sending DeleteTablet for 3 replicas of tablet e75bc2fdf31741c4ba745872c1ad13f5
I20260929 11:21:24.395937 8135 tablet_service.cc:1514] Processing DeleteTablet for tablet e75bc2fdf31741c4ba745872c1ad13f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54476
I20260929 11:21:24.395938 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet e75bc2fdf31741c4ba745872c1ad13f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35342
I20260929 11:21:24.396829 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet e75bc2fdf31741c4ba745872c1ad13f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40648
I20260929 11:21:24.397380 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:24.398321 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:24.401087 8533 tablet_replica.cc:330] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: stopping tablet replica
I20260929 11:21:24.401614 8533 raft_consensus.cc:2229] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:24.402210 8533 raft_consensus.cc:2258] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:24.402868 8531 tablet_replica.cc:330] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: stopping tablet replica
I20260929 11:21:24.404399 8531 raft_consensus.cc:2229] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:24.405270 8531 raft_consensus.cc:2258] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:24.406153 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 795a34a186244777bd9018cc17167227 (DEFAULT_TABLE table=test-table [id=88e9fb5f281b4c0ea99cee1bfbb73cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:24.406694 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795a34a186244777bd9018cc17167227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:24.407711 8533 ts_tablet_manager.cc:1905] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:24.407750 8531 ts_tablet_manager.cc:1905] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:24.413338 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 795a34a186244777bd9018cc17167227 (DEFAULT_TABLE table=test-table [id=88e9fb5f281b4c0ea99cee1bfbb73cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:24.413807 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795a34a186244777bd9018cc17167227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:24.417148 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 795a34a186244777bd9018cc17167227 (DEFAULT_TABLE table=test-table [id=88e9fb5f281b4c0ea99cee1bfbb73cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:24.417374 8531 ts_tablet_manager.cc:1918] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:24.417536 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795a34a186244777bd9018cc17167227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:24.417660 8531 log.cc:1196] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/e75bc2fdf31741c4ba745872c1ad13f5
I20260929 11:21:24.418419 8531 ts_tablet_manager.cc:1939] T e75bc2fdf31741c4ba745872c1ad13f5 P a36610f683fe44548b1c09af2c7e34fa: Deleting consensus metadata
I20260929 11:21:24.420578 8532 tablet_replica.cc:330] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: stopping tablet replica
I20260929 11:21:24.420666 7982 catalog_manager.cc:4841] TS a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259): tablet e75bc2fdf31741c4ba745872c1ad13f5 (table test-table [id=907a2a9feaf2480095642753bfec7ff5]) successfully deleted
I20260929 11:21:24.421044 8532 raft_consensus.cc:2229] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:24.421557 8532 raft_consensus.cc:2258] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:24.423133 8532 ts_tablet_manager.cc:1905] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:24.423368 8533 ts_tablet_manager.cc:1918] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:24.423529 8533 log.cc:1196] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/e75bc2fdf31741c4ba745872c1ad13f5
I20260929 11:21:24.424026 8533 ts_tablet_manager.cc:1939] T e75bc2fdf31741c4ba745872c1ad13f5 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting consensus metadata
I20260929 11:21:24.425662 7982 catalog_manager.cc:4841] TS 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865): tablet e75bc2fdf31741c4ba745872c1ad13f5 (table test-table [id=907a2a9feaf2480095642753bfec7ff5]) successfully deleted
I20260929 11:21:24.426121 8509 tablet_bootstrap.cc:492] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Bootstrap starting.
I20260929 11:21:24.426256 8508 tablet_bootstrap.cc:492] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Bootstrap starting.
I20260929 11:21:24.427695 8509 tablet_bootstrap.cc:654] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:24.427767 8508 tablet_bootstrap.cc:654] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:24.432724 8507 tablet_bootstrap.cc:492] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Bootstrap starting.
I20260929 11:21:24.435468 8532 ts_tablet_manager.cc:1918] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:24.435612 8532 log.cc:1196] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/e75bc2fdf31741c4ba745872c1ad13f5
I20260929 11:21:24.436138 8532 ts_tablet_manager.cc:1939] T e75bc2fdf31741c4ba745872c1ad13f5 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting consensus metadata
I20260929 11:21:24.436689 8507 tablet_bootstrap.cc:654] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:24.437789 7984 catalog_manager.cc:4841] TS 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141): tablet e75bc2fdf31741c4ba745872c1ad13f5 (table test-table [id=907a2a9feaf2480095642753bfec7ff5]) successfully deleted
I20260929 11:21:24.443063 8509 tablet_bootstrap.cc:492] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: No bootstrap required, opened a new log
I20260929 11:21:24.443207 8509 ts_tablet_manager.cc:1397] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260929 11:21:24.443955 8509 raft_consensus.cc:348] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.444177 8509 raft_consensus.cc:374] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:24.444262 8509 raft_consensus.cc:729] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Initialized, Role: FOLLOWER
I20260929 11:21:24.444553 8507 tablet_bootstrap.cc:492] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: No bootstrap required, opened a new log
I20260929 11:21:24.444555 8509 consensus_queue.cc:260] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.444762 8507 ts_tablet_manager.cc:1397] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260929 11:21:24.445812 8507 raft_consensus.cc:348] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.446094 8507 raft_consensus.cc:374] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:24.446203 8507 raft_consensus.cc:729] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Initialized, Role: FOLLOWER
I20260929 11:21:24.446537 8507 consensus_queue.cc:260] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.447264 8507 ts_tablet_manager.cc:1428] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:21:24.448065 8509 ts_tablet_manager.cc:1428] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 11:21:24.449093 8508 tablet_bootstrap.cc:492] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: No bootstrap required, opened a new log
I20260929 11:21:24.449271 8508 ts_tablet_manager.cc:1397] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.006s
I20260929 11:21:24.450083 8508 raft_consensus.cc:348] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.450307 8508 raft_consensus.cc:374] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:24.450377 8508 raft_consensus.cc:729] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691fe906ef774e7b87a73ff24ec3e58b, State: Initialized, Role: FOLLOWER
I20260929 11:21:24.450599 8508 consensus_queue.cc:260] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.451241 8508 ts_tablet_manager.cc:1428] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 11:21:24.584379 8513 raft_consensus.cc:482] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:24.584646 8513 raft_consensus.cc:504] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.586537 8513 leader_election.cc:288] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865), a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:24.594417 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.586961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51459 (local address 127.0.97.2:44865)
0929 11:21:24.587147 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.587153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.587521 (+ 368us) server_negotiation.cc:407] Connection header received
0929 11:21:24.587847 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.587856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.587954 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.588045 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:24.588849 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.589981 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.592280 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.593367 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.593499 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.593632 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.594019 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.594142 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:21:24.594455 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.586845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44865 (local address 127.0.97.1:51459)
0929 11:21:24.587420 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.587439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.587630 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.588241 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.588244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.588265 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:24.588688 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.588701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.590111 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.590115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.592124 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.592141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.593625 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.593632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.593844 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.593882 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.593993 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.594132 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
I20260929 11:21:24.595341 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795a34a186244777bd9018cc17167227" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b" is_pre_election: true
I20260929 11:21:24.595633 8295 raft_consensus.cc:2454] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 0.
I20260929 11:21:24.596091 8085 leader_election.cc:302] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, 691fe906ef774e7b87a73ff24ec3e58b; no voters:
I20260929 11:21:24.596477 8513 raft_consensus.cc:2781] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:24.596629 8513 raft_consensus.cc:482] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:24.596741 8513 raft_consensus.cc:3037] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.596768 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.587530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.97.1:44825)
0929 11:21:24.588295 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.588313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.588461 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.590158 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.590163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.590191 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:24.590569 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.590580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.591969 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.591974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.594162 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.594175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.595916 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.595921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.596087 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.596117 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.596195 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.596318 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":1,"thread_start_us":459,"threads_started":1}
I20260929 11:21:24.597054 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.587618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44825 (local address 127.0.97.3:41259)
0929 11:21:24.587822 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.587828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.588429 (+ 601us) server_negotiation.cc:407] Connection header received
0929 11:21:24.588631 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.588637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.588711 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.588804 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:24.590754 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.591851 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.594354 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.595567 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.595657 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.595749 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.596318 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.596454 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:21:24.597790 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795a34a186244777bd9018cc17167227" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36610f683fe44548b1c09af2c7e34fa" is_pre_election: true
I20260929 11:21:24.598251 8433 raft_consensus.cc:2454] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 0.
I20260929 11:21:24.599351 8513 raft_consensus.cc:504] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.599910 8513 leader_election.cc:288] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 election: Requested vote from peers 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865), a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:24.600334 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795a34a186244777bd9018cc17167227" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b"
I20260929 11:21:24.600529 8295 raft_consensus.cc:3037] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.600670 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795a34a186244777bd9018cc17167227" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36610f683fe44548b1c09af2c7e34fa"
I20260929 11:21:24.600847 8433 raft_consensus.cc:3037] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.602548 8295 raft_consensus.cc:2454] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 1.
I20260929 11:21:24.602952 8085 leader_election.cc:302] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, 691fe906ef774e7b87a73ff24ec3e58b; no voters:
I20260929 11:21:24.603240 8513 raft_consensus.cc:2781] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:24.603745 8513 raft_consensus.cc:686] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 LEADER]: Becoming Leader. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Running, Role: LEADER
I20260929 11:21:24.604014 8433 raft_consensus.cc:2454] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 1.
I20260929 11:21:24.604391 8513 consensus_queue.cc:237] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.607599 7998 catalog_manager.cc:5495] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "403e3f833f7e4fb28ccdc432393b9d90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:21:24 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:24.654920 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.639313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41243 (local address 127.0.0.1:40778)
0929 11:21:24.639965 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.639987 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.640145 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.640539 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.640544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.641419 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:24.641694 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.641706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.642784 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.642789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.644224 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.644240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.644978 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.644987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.645159 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.645835 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:24.645867 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:24.648344 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:24.652375 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:24.652384 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:24.652389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:24.652904 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:24.653277 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:24.653281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:24.653283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:24.653457 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:24.654118 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:24.654124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:24.654263 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.654400 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.654607 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":302,"threads_started":1}
I20260929 11:21:24.655233 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.639313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.97.62:41243)
0929 11:21:24.640263 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.640267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.640286 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:21:24.640328 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.640331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.640380 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.640467 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:24.641850 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.642672 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.644350 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.644818 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.644906 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.648547 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:24.648556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:24.648561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:24.648615 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:24.652227 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:24.653007 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:24.653012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:24.653015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:24.653173 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:24.653600 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:24.653605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:24.653608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:24.653961 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:24.654161 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.654527 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.654745 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":736,"threads_started":1}
I20260929 11:21:24.660550 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:21:24.661670 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:24.668701 8134 tablet_service.cc:1467] Processing CreateTablet for tablet d15fb051e0aa46048b606b46d6f62982 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=887d7628a6de43249e951b3814a03d99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:21:24.669024 8274 tablet_service.cc:1467] Processing CreateTablet for tablet d15fb051e0aa46048b606b46d6f62982 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=887d7628a6de43249e951b3814a03d99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:21:24.669275 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15fb051e0aa46048b606b46d6f62982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:24.669178 8413 tablet_service.cc:1467] Processing CreateTablet for tablet d15fb051e0aa46048b606b46d6f62982 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=887d7628a6de43249e951b3814a03d99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:21:24.669584 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15fb051e0aa46048b606b46d6f62982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:24.669689 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15fb051e0aa46048b606b46d6f62982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:24.676769 8508 tablet_bootstrap.cc:492] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: Bootstrap starting.
I20260929 11:21:24.677554 8509 tablet_bootstrap.cc:492] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: Bootstrap starting.
I20260929 11:21:24.678472 8508 tablet_bootstrap.cc:654] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:24.678821 8509 tablet_bootstrap.cc:654] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:24.680750 8507 tablet_bootstrap.cc:492] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: Bootstrap starting.
I20260929 11:21:24.682085 8507 tablet_bootstrap.cc:654] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:24.683630 8509 tablet_bootstrap.cc:492] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: No bootstrap required, opened a new log
I20260929 11:21:24.683805 8509 ts_tablet_manager.cc:1397] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260929 11:21:24.684363 8509 raft_consensus.cc:348] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.684607 8509 raft_consensus.cc:374] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:24.684707 8509 raft_consensus.cc:729] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Initialized, Role: FOLLOWER
I20260929 11:21:24.684942 8509 consensus_queue.cc:260] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.685343 8513 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.685525 8509 ts_tablet_manager.cc:1428] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:21:24.685629 8513 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.689074 8508 tablet_bootstrap.cc:492] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: No bootstrap required, opened a new log
I20260929 11:21:24.689270 8508 ts_tablet_manager.cc:1397] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260929 11:21:24.689970 8508 raft_consensus.cc:348] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.690186 8508 raft_consensus.cc:374] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:24.690253 8508 raft_consensus.cc:729] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691fe906ef774e7b87a73ff24ec3e58b, State: Initialized, Role: FOLLOWER
I20260929 11:21:24.690490 8508 consensus_queue.cc:260] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.691006 8515 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.691522 8515 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.691076 8508 ts_tablet_manager.cc:1428] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:21:24.695503 8507 tablet_bootstrap.cc:492] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: No bootstrap required, opened a new log
I20260929 11:21:24.695698 8507 ts_tablet_manager.cc:1397] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260929 11:21:24.696388 8507 raft_consensus.cc:348] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.696611 8507 raft_consensus.cc:374] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:24.696718 8507 raft_consensus.cc:729] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Initialized, Role: FOLLOWER
I20260929 11:21:24.696942 8507 consensus_queue.cc:260] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.697474 8507 ts_tablet_manager.cc:1428] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:21:24.697335 8518 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.697650 8518 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.728268 8518 raft_consensus.cc:482] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:24.728489 8518 raft_consensus.cc:504] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.729190 8518 leader_election.cc:288] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141), 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865)
I20260929 11:21:24.729708 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15fb051e0aa46048b606b46d6f62982" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403e3f833f7e4fb28ccdc432393b9d90" is_pre_election: true
I20260929 11:21:24.729748 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15fb051e0aa46048b606b46d6f62982" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b" is_pre_election: true
I20260929 11:21:24.729935 8156 raft_consensus.cc:2454] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 0.
I20260929 11:21:24.730049 8295 raft_consensus.cc:2454] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 0.
I20260929 11:21:24.730399 8366 leader_election.cc:302] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, a36610f683fe44548b1c09af2c7e34fa; no voters:
I20260929 11:21:24.730686 8518 raft_consensus.cc:2781] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:24.730795 8518 raft_consensus.cc:482] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:24.730901 8518 raft_consensus.cc:3037] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.733225 8518 raft_consensus.cc:504] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.733814 8518 leader_election.cc:288] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 election: Requested vote from peers 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141), 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865)
I20260929 11:21:24.734273 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15fb051e0aa46048b606b46d6f62982" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403e3f833f7e4fb28ccdc432393b9d90"
I20260929 11:21:24.734265 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d15fb051e0aa46048b606b46d6f62982" candidate_uuid: "a36610f683fe44548b1c09af2c7e34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b"
I20260929 11:21:24.734477 8156 raft_consensus.cc:3037] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.734503 8295 raft_consensus.cc:3037] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:24.736617 8156 raft_consensus.cc:2454] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 1.
I20260929 11:21:24.737066 8366 leader_election.cc:302] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, a36610f683fe44548b1c09af2c7e34fa; no voters:
I20260929 11:21:24.737306 8518 raft_consensus.cc:2781] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:24.737553 8518 raft_consensus.cc:686] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [term 1 LEADER]: Becoming Leader. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Running, Role: LEADER
I20260929 11:21:24.737542 8295 raft_consensus.cc:2454] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36610f683fe44548b1c09af2c7e34fa in term 1.
I20260929 11:21:24.737841 8518 consensus_queue.cc:237] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:24.739637 8529 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: TxnStatusTablet state changed. Reason: New leader a36610f683fe44548b1c09af2c7e34fa. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.740034 8529 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:21:24.740897 8552 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:21:24.741142 8552 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:21:24.742022 7997 catalog_manager.cc:5495] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa reported cstate change: term changed from 0 to 1, leader changed from <none> to a36610f683fe44548b1c09af2c7e34fa (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } health_report { overall_health: HEALTHY } } }
I20260929 11:21:24.770489 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.764036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.97.3:41259)
0929 11:21:24.764209 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.764215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.764233 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:21:24.764486 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.764492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.764577 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.764690 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:24.765107 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.766258 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.768314 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.768985 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.769159 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.769644 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.769749 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.769857 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.770010 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.770112 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:21:24.772787 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.763923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.0.1:35372)
0929 11:21:24.764133 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.764156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.764295 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.764735 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.764741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.764754 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:21:24.764942 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.764952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.766410 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.766418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.768169 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.768184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.769187 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.769194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.769393 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.769471 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.769858 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.769881 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.769975 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.770132 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:21:24.776962 8156 raft_consensus.cc:1261] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Refusing update from remote peer a36610f683fe44548b1c09af2c7e34fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:24.777128 8295 raft_consensus.cc:1261] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Refusing update from remote peer a36610f683fe44548b1c09af2c7e34fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:24.777565 8529 consensus_queue.cc:1035] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:24.778487 8529 consensus_queue.cc:1035] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:24.790542 8513 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: TxnStatusTablet state changed. Reason: New leader a36610f683fe44548b1c09af2c7e34fa. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.790805 8513 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.792037 8515 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: TxnStatusTablet state changed. Reason: New leader a36610f683fe44548b1c09af2c7e34fa. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.792330 8515 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.793483 8518 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.793722 8518 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:21:24.799866 8514 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.800163 8514 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P a36610f683fe44548b1c09af2c7e34fa: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:21:24.802726 8515 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.802975 8515 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P 691fe906ef774e7b87a73ff24ec3e58b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.803753 8513 tablet_replica.cc:439] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a36610f683fe44548b1c09af2c7e34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } }
I20260929 11:21:24.804003 8513 tablet_replica.cc:442] T d15fb051e0aa46048b606b46d6f62982 P 403e3f833f7e4fb28ccdc432393b9d90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:21:24.815068 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.802584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.0.97.1:34141)
0929 11:21:24.805028 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.805035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.805056 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:21:24.805149 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.805154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.805245 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.806426 (+ 1181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:24.806594 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.807673 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.812733 (+ 5060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.813118 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.813271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.813812 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.814366 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.814472 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.814627 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.814710 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2318,"thread_start_us":5908,"threads_started":1}
I20260929 11:21:24.831312 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.825837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40527 (local address 127.0.97.3:41259)
0929 11:21:24.826088 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.826094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.826366 (+ 272us) server_negotiation.cc:407] Connection header received
0929 11:21:24.826581 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.826586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.826661 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.826762 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:24.827237 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.828038 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.829494 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.829961 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.830120 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.830522 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.830655 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.830746 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.830885 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.830963 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:21:24.831367 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.825734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.97.1:40527)
0929 11:21:24.826263 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.826279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.826438 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.826774 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.826778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.826789 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:21:24.827096 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.827104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.828171 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.828176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.829344 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.829356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.830152 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.830158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.830289 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.830340 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.830761 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.830775 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.830865 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.830993 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
I20260929 11:21:24.841579 8296 raft_consensus.cc:1261] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Refusing update from remote peer 403e3f833f7e4fb28ccdc432393b9d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:24.842051 8433 raft_consensus.cc:1261] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Refusing update from remote peer 403e3f833f7e4fb28ccdc432393b9d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:24.842315 8539 consensus_queue.cc:1035] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:24.842697 8513 consensus_queue.cc:1035] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:24.879002 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.872467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39349 (local address 127.0.97.1:34141)
0929 11:21:24.872686 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.872694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.873363 (+ 669us) server_negotiation.cc:407] Connection header received
0929 11:21:24.873577 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.873583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.873684 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.873767 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:24.874378 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.875543 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.877072 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.877743 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.877862 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.878423 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.878521 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.878596 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:24.878728 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 11:21:24.878785 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:21:24.879329 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.872514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34141 (local address 127.0.97.3:39349)
0929 11:21:24.873267 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:24.873284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:24.873433 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:24.873795 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:24.873801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:24.873815 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:21:24.874211 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.874225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.875667 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.875671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.876933 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:24.876946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.878016 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.878023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:24.878200 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.878263 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:21:24.878618 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:21:24.878628 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:21:24.878703 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 11:21:24.878838 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":387,"threads_started":1}
I20260929 11:21:24.914681 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:24.914971 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:24.918479 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Sending DeleteTablet for 3 replicas of tablet 795a34a186244777bd9018cc17167227
I20260929 11:21:24.919489 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 795a34a186244777bd9018cc17167227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35342
I20260929 11:21:24.919757 8531 tablet_replica.cc:330] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: stopping tablet replica
I20260929 11:21:24.919948 8531 raft_consensus.cc:2229] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:24.919981 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 795a34a186244777bd9018cc17167227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40648
I20260929 11:21:24.920138 8531 raft_consensus.cc:2258] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:24.920297 8533 tablet_replica.cc:330] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: stopping tablet replica
I20260929 11:21:24.920495 8533 raft_consensus.cc:2229] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:24.920719 8533 raft_consensus.cc:2258] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:24.920693 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 795a34a186244777bd9018cc17167227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54476
I20260929 11:21:24.920933 8532 tablet_replica.cc:330] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: stopping tablet replica
I20260929 11:21:24.921144 8532 raft_consensus.cc:2229] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:24.921593 8532 raft_consensus.cc:2258] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:24.921869 8533 ts_tablet_manager.cc:1905] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:24.922713 8532 ts_tablet_manager.cc:1905] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:24.928435 8533 ts_tablet_manager.cc:1918] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:21:24.928576 8533 log.cc:1196] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/795a34a186244777bd9018cc17167227
I20260929 11:21:24.928994 8533 ts_tablet_manager.cc:1939] T 795a34a186244777bd9018cc17167227 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting consensus metadata
I20260929 11:21:24.929028 8532 ts_tablet_manager.cc:1918] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:21:24.929171 8532 log.cc:1196] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/795a34a186244777bd9018cc17167227
I20260929 11:21:24.929530 8532 ts_tablet_manager.cc:1939] T 795a34a186244777bd9018cc17167227 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting consensus metadata
I20260929 11:21:24.930094 7982 catalog_manager.cc:4841] TS 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865): tablet 795a34a186244777bd9018cc17167227 (table test-table [id=88e9fb5f281b4c0ea99cee1bfbb73cec]) successfully deleted
I20260929 11:21:24.931175 8531 ts_tablet_manager.cc:1905] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:24.931214 7984 catalog_manager.cc:4841] TS 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141): tablet 795a34a186244777bd9018cc17167227 (table test-table [id=88e9fb5f281b4c0ea99cee1bfbb73cec]) successfully deleted
I20260929 11:21:24.938266 8531 ts_tablet_manager.cc:1918] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:21:24.938439 8531 log.cc:1196] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/795a34a186244777bd9018cc17167227
I20260929 11:21:24.938891 8531 ts_tablet_manager.cc:1939] T 795a34a186244777bd9018cc17167227 P a36610f683fe44548b1c09af2c7e34fa: Deleting consensus metadata
I20260929 11:21:24.940215 7982 catalog_manager.cc:4841] TS a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259): tablet 795a34a186244777bd9018cc17167227 (table test-table [id=88e9fb5f281b4c0ea99cee1bfbb73cec]) successfully deleted
Sep 29 11:21:24 dist-test-slave-srst krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:21:24 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:24 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:21:24.989108 8546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:24.975538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.0.97.62:41243)
0929 11:21:24.975745 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.975751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.977185 (+ 1434us) server_negotiation.cc:407] Connection header received
0929 11:21:24.977370 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.977376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.977488 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.977622 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:24.979266 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.980492 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.982320 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.982951 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.983089 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:24.988842 (+ 5753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40780: BlockingRecv error: recv got EOF from 127.0.0.1:40780 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 11:21:24 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 11:21:25.009660 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:24.992324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.0.97.62:41243)
0929 11:21:24.992487 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:24.992493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:24.993018 (+ 525us) server_negotiation.cc:407] Connection header received
0929 11:21:24.993320 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:24.993325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:24.993401 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:24.993499 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:24.995044 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.996124 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.997949 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:24.998618 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:24.998744 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.003405 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:25.003412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:25.003417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:25.003455 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:25.006810 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.007545 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.007550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.007554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.007696 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.008140 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.008145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.008149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.008497 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:25.008768 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.009081 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.009293 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:21:25.013684 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:25.014425 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:25.021867 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 0e524e9eac014f64a534219f2df7e755 (DEFAULT_TABLE table=test-table [id=2a2c2a02e5d64aca8d76be068fdb683e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:25.022353 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e524e9eac014f64a534219f2df7e755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:25.029055 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 0e524e9eac014f64a534219f2df7e755 (DEFAULT_TABLE table=test-table [id=2a2c2a02e5d64aca8d76be068fdb683e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:25.029242 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 0e524e9eac014f64a534219f2df7e755 (DEFAULT_TABLE table=test-table [id=2a2c2a02e5d64aca8d76be068fdb683e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:25.029480 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e524e9eac014f64a534219f2df7e755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:25.029660 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e524e9eac014f64a534219f2df7e755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:25.036576 8508 tablet_bootstrap.cc:492] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Bootstrap starting.
I20260929 11:21:25.038236 8508 tablet_bootstrap.cc:654] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:25.038713 8509 tablet_bootstrap.cc:492] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Bootstrap starting.
I20260929 11:21:25.039640 8507 tablet_bootstrap.cc:492] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Bootstrap starting.
I20260929 11:21:25.040318 8509 tablet_bootstrap.cc:654] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:25.041395 8507 tablet_bootstrap.cc:654] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:25.049240 8508 tablet_bootstrap.cc:492] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: No bootstrap required, opened a new log
I20260929 11:21:25.049434 8508 ts_tablet_manager.cc:1397] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20260929 11:21:25.050482 8508 raft_consensus.cc:348] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.050859 8508 raft_consensus.cc:374] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:25.050968 8508 raft_consensus.cc:729] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691fe906ef774e7b87a73ff24ec3e58b, State: Initialized, Role: FOLLOWER
I20260929 11:21:25.051285 8508 consensus_queue.cc:260] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.052884 8508 ts_tablet_manager.cc:1428] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:25.057991 8507 tablet_bootstrap.cc:492] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: No bootstrap required, opened a new log
I20260929 11:21:25.058209 8507 ts_tablet_manager.cc:1397] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20260929 11:21:25.059242 8507 raft_consensus.cc:348] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.059593 8507 raft_consensus.cc:374] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:25.059726 8507 raft_consensus.cc:729] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Initialized, Role: FOLLOWER
I20260929 11:21:25.060056 8507 consensus_queue.cc:260] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.063007 8507 ts_tablet_manager.cc:1428] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260929 11:21:25.066977 8509 tablet_bootstrap.cc:492] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: No bootstrap required, opened a new log
I20260929 11:21:25.067165 8509 ts_tablet_manager.cc:1397] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.008s
I20260929 11:21:25.068130 8509 raft_consensus.cc:348] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.068423 8509 raft_consensus.cc:374] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:25.068528 8509 raft_consensus.cc:729] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Initialized, Role: FOLLOWER
I20260929 11:21:25.068802 8509 consensus_queue.cc:260] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.069841 8509 ts_tablet_manager.cc:1428] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:25.119411 8513 raft_consensus.cc:482] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:25.119608 8513 raft_consensus.cc:504] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.120240 8513 leader_election.cc:288] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865), a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:25.121130 8296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e524e9eac014f64a534219f2df7e755" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b" is_pre_election: true
I20260929 11:21:25.121359 8296 raft_consensus.cc:2454] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 0.
I20260929 11:21:25.121923 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e524e9eac014f64a534219f2df7e755" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36610f683fe44548b1c09af2c7e34fa" is_pre_election: true
I20260929 11:21:25.122145 8433 raft_consensus.cc:2454] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 0.
I20260929 11:21:25.122836 8085 leader_election.cc:302] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, a36610f683fe44548b1c09af2c7e34fa; no voters:
I20260929 11:21:25.123396 8513 raft_consensus.cc:2781] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:25.123528 8513 raft_consensus.cc:482] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:25.123627 8513 raft_consensus.cc:3037] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:25.126931 8513 raft_consensus.cc:504] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.128654 8296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e524e9eac014f64a534219f2df7e755" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fe906ef774e7b87a73ff24ec3e58b"
I20260929 11:21:25.128876 8296 raft_consensus.cc:3037] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:25.131598 8296 raft_consensus.cc:2454] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 1.
I20260929 11:21:25.132028 8085 leader_election.cc:302] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, 691fe906ef774e7b87a73ff24ec3e58b; no voters:
I20260929 11:21:25.132294 8562 raft_consensus.cc:2781] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:25.132432 8562 raft_consensus.cc:686] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 LEADER]: Becoming Leader. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Running, Role: LEADER
I20260929 11:21:25.132783 8562 consensus_queue.cc:237] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.133076 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e524e9eac014f64a534219f2df7e755" candidate_uuid: "403e3f833f7e4fb28ccdc432393b9d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36610f683fe44548b1c09af2c7e34fa"
I20260929 11:21:25.133316 8433 raft_consensus.cc:3037] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:25.137857 8433 raft_consensus.cc:2454] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403e3f833f7e4fb28ccdc432393b9d90 in term 1.
I20260929 11:21:25.138458 8513 leader_election.cc:288] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [CANDIDATE]: Term 1 election: Requested vote from peers 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865), a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:25.141415 7998 catalog_manager.cc:5495] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "403e3f833f7e4fb28ccdc432393b9d90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20260929 11:21:25.156782 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.151559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.0.97.1:34141)
0929 11:21:25.151724 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.151730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.151848 (+ 118us) server_negotiation.cc:407] Connection header received
0929 11:21:25.152163 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.152166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.152283 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.152379 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:25.152800 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.153637 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.155059 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.155508 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.155643 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.156027 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:25.156135 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:25.156229 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.156375 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.156449 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:21:25.159374 8296 raft_consensus.cc:1261] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Refusing update from remote peer 403e3f833f7e4fb28ccdc432393b9d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:25.159363 8433 raft_consensus.cc:1261] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Refusing update from remote peer 403e3f833f7e4fb28ccdc432393b9d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:25.160144 8562 consensus_queue.cc:1035] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:25.160447 8513 consensus_queue.cc:1035] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:25.180303 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:25.180541 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:25.184737 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Sending DeleteTablet for 3 replicas of tablet 0e524e9eac014f64a534219f2df7e755
I20260929 11:21:25.185685 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e524e9eac014f64a534219f2df7e755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35342
I20260929 11:21:25.185945 8531 tablet_replica.cc:330] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: stopping tablet replica
I20260929 11:21:25.186156 8531 raft_consensus.cc:2229] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:25.186401 8531 raft_consensus.cc:2258] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:25.186887 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e524e9eac014f64a534219f2df7e755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54476
I20260929 11:21:25.187191 8532 tablet_replica.cc:330] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: stopping tablet replica
I20260929 11:21:25.187351 8532 raft_consensus.cc:2229] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:25.187472 8531 ts_tablet_manager.cc:1905] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:25.188010 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e524e9eac014f64a534219f2df7e755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35342
I20260929 11:21:25.188526 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e524e9eac014f64a534219f2df7e755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40648
I20260929 11:21:25.188712 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e524e9eac014f64a534219f2df7e755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54476
I20260929 11:21:25.188805 8533 tablet_replica.cc:330] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: stopping tablet replica
I20260929 11:21:25.188958 8533 raft_consensus.cc:2229] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:25.189122 8533 pending_rounds.cc:62] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Trying to abort 2 pending ops.
I20260929 11:21:25.189209 8533 pending_rounds.cc:68] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334628905606705152 op_type: WRITE_OP write_request { tablet_id: "0e524e9eac014f64a534219f2df7e755" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\341\300\356\325\006\"8\n\ntest-admin\022*\n 2a2c2a02e5d64aca8d76be068fdb683e\020\001\030\001 \001(\001" signature: "<redacted>""m\320\230\034\365\036\241\336\265\227\356\323i.\206\272X\302\240\206=\260\002\037\252tq\345r\220\272W\004\334\305\207\364=Arwt\302\227N\356q6+v\272\213\242U\376d\310\264\"7zBr\223" signing_key_seq_num: 0 } } request_id { client_id: "49cde704142c4c5c8e9b4f21dd5dd948" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260929 11:21:25.189553 8532 raft_consensus.cc:2258] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:25.189797 8533 pending_rounds.cc:68] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334628905506426880 op_type: NO_OP noop_request { }
I20260929 11:21:25.189913 8533 raft_consensus.cc:2866] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 11:21:25.190062 8533 raft_consensus.cc:2258] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:25.191107 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:25.191713 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:25.196043 8531 ts_tablet_manager.cc:1918] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:25.196259 8531 log.cc:1196] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/0e524e9eac014f64a534219f2df7e755
I20260929 11:21:25.196851 8531 ts_tablet_manager.cc:1939] T 0e524e9eac014f64a534219f2df7e755 P a36610f683fe44548b1c09af2c7e34fa: Deleting consensus metadata
I20260929 11:21:25.198020 7982 catalog_manager.cc:4841] TS a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259): tablet 0e524e9eac014f64a534219f2df7e755 (table test-table [id=2a2c2a02e5d64aca8d76be068fdb683e]) successfully deleted
I20260929 11:21:25.199213 8274 tablet_service.cc:1467] Processing CreateTablet for tablet fce8d42438f04a47b4129c017d6d135a (DEFAULT_TABLE table=test-table [id=5bae06f9997245c1b93f3f09a067144b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:25.199504 8134 tablet_service.cc:1467] Processing CreateTablet for tablet fce8d42438f04a47b4129c017d6d135a (DEFAULT_TABLE table=test-table [id=5bae06f9997245c1b93f3f09a067144b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:25.199633 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce8d42438f04a47b4129c017d6d135a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:25.199900 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce8d42438f04a47b4129c017d6d135a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:25.205669 8413 tablet_service.cc:1467] Processing CreateTablet for tablet fce8d42438f04a47b4129c017d6d135a (DEFAULT_TABLE table=test-table [id=5bae06f9997245c1b93f3f09a067144b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:25.206064 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce8d42438f04a47b4129c017d6d135a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:25.207234 8509 tablet_bootstrap.cc:492] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Bootstrap starting.
I20260929 11:21:25.209769 8509 tablet_bootstrap.cc:654] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:25.214669 8508 tablet_bootstrap.cc:492] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Bootstrap starting.
I20260929 11:21:25.217089 8508 tablet_bootstrap.cc:654] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:25.217305 8507 tablet_bootstrap.cc:492] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Bootstrap starting.
I20260929 11:21:25.217607 8533 ts_tablet_manager.cc:1905] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:25.219049 8507 tablet_bootstrap.cc:654] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Neither blocks nor log segments found. Creating new log.
W20260929 11:21:25.224956 7982 catalog_manager.cc:4808] TS a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259): delete failed for tablet 0e524e9eac014f64a534219f2df7e755 because the tablet was not found. No further retry: Not found: Tablet not found: 0e524e9eac014f64a534219f2df7e755
I20260929 11:21:25.226359 8508 tablet_bootstrap.cc:492] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: No bootstrap required, opened a new log
I20260929 11:21:25.226506 8532 ts_tablet_manager.cc:1905] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:25.226545 8508 ts_tablet_manager.cc:1397] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260929 11:21:25.227289 8508 raft_consensus.cc:348] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.227514 8508 raft_consensus.cc:374] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:25.227576 8508 raft_consensus.cc:729] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691fe906ef774e7b87a73ff24ec3e58b, State: Initialized, Role: FOLLOWER
I20260929 11:21:25.228912 8508 consensus_queue.cc:260] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.229528 8508 ts_tablet_manager.cc:1428] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 11:21:25.229750 8533 ts_tablet_manager.cc:1918] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:25.229872 8533 log.cc:1196] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/0e524e9eac014f64a534219f2df7e755
I20260929 11:21:25.230296 8533 ts_tablet_manager.cc:1939] T 0e524e9eac014f64a534219f2df7e755 P 691fe906ef774e7b87a73ff24ec3e58b: Deleting consensus metadata
I20260929 11:21:25.231339 7982 catalog_manager.cc:4841] TS 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865): tablet 0e524e9eac014f64a534219f2df7e755 (table test-table [id=2a2c2a02e5d64aca8d76be068fdb683e]) successfully deleted
I20260929 11:21:25.234459 8532 ts_tablet_manager.cc:1918] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:25.234709 8532 log.cc:1196] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/0e524e9eac014f64a534219f2df7e755
I20260929 11:21:25.235342 8532 ts_tablet_manager.cc:1939] T 0e524e9eac014f64a534219f2df7e755 P 403e3f833f7e4fb28ccdc432393b9d90: Deleting consensus metadata
I20260929 11:21:25.236827 7984 catalog_manager.cc:4841] TS 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141): tablet 0e524e9eac014f64a534219f2df7e755 (table test-table [id=2a2c2a02e5d64aca8d76be068fdb683e]) successfully deleted
W20260929 11:21:25.237231 7984 catalog_manager.cc:4808] TS 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141): delete failed for tablet 0e524e9eac014f64a534219f2df7e755 because the tablet was not found. No further retry: Not found: Tablet not found: 0e524e9eac014f64a534219f2df7e755
I20260929 11:21:25.244028 8515 raft_consensus.cc:482] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:25.244228 8515 raft_consensus.cc:504] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.245201 8509 tablet_bootstrap.cc:492] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: No bootstrap required, opened a new log
I20260929 11:21:25.245420 8509 ts_tablet_manager.cc:1397] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.000s
I20260929 11:21:25.245980 8515 leader_election.cc:288] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141), a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:25.246451 8509 raft_consensus.cc:348] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.246788 8509 raft_consensus.cc:374] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:25.246927 8509 raft_consensus.cc:729] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403e3f833f7e4fb28ccdc432393b9d90, State: Initialized, Role: FOLLOWER
I20260929 11:21:25.247292 8509 consensus_queue.cc:260] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.247848 8509 ts_tablet_manager.cc:1428] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 11:21:25.256697 8507 tablet_bootstrap.cc:492] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: No bootstrap required, opened a new log
I20260929 11:21:25.257009 8507 ts_tablet_manager.cc:1397] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.001s
I20260929 11:21:25.258131 8507 raft_consensus.cc:348] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.258530 8507 raft_consensus.cc:374] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:25.258661 8507 raft_consensus.cc:729] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36610f683fe44548b1c09af2c7e34fa, State: Initialized, Role: FOLLOWER
I20260929 11:21:25.259025 8507 consensus_queue.cc:260] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.263429 8507 ts_tablet_manager.cc:1428] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 11:21:25.274539 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.247267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51449 (local address 127.0.97.1:34141)
0929 11:21:25.247636 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.247643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.266750 (+ 19107us) server_negotiation.cc:407] Connection header received
0929 11:21:25.266965 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.266972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.267084 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.267180 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:25.267917 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.269083 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.271651 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.273295 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.273427 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.273543 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.274086 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.274215 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260929 11:21:25.275276 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.246392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34141 (local address 127.0.97.2:51449)
0929 11:21:25.266636 (+ 20244us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:25.266663 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:25.266829 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:25.267377 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:25.267381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:25.267413 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:25.267727 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.267738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.269262 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.269267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.271486 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.271499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.273722 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.273729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.273898 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.273933 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:21:25.274802 (+ 869us) client_negotiation.cc:240] Negotiation successful
0929 11:21:25.274934 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20109,"spinlock_wait_cycles":11264,"threads_started":1}
I20260929 11:21:25.276044 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce8d42438f04a47b4129c017d6d135a" candidate_uuid: "691fe906ef774e7b87a73ff24ec3e58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403e3f833f7e4fb28ccdc432393b9d90" is_pre_election: true
I20260929 11:21:25.276295 8156 raft_consensus.cc:2454] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 691fe906ef774e7b87a73ff24ec3e58b in term 0.
I20260929 11:21:25.276453 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.247430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56597 (local address 127.0.97.3:41259)
0929 11:21:25.247772 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.247778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.267458 (+ 19680us) server_negotiation.cc:407] Connection header received
0929 11:21:25.267744 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.267748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.267824 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.267903 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:25.268854 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.269961 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.272328 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.274895 (+ 2567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.274999 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.275124 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.276054 (+ 930us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.276202 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260929 11:21:25.277079 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.246788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.97.2:56597)
0929 11:21:25.267377 (+ 20589us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:25.267393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:25.267559 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:25.268173 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:25.268178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:25.268205 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:25.268648 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.268660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.270231 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.270236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.272180 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.272192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.275692 (+ 3500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.275701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.275872 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.275907 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:21:25.276774 (+ 867us) client_negotiation.cc:240] Negotiation successful
0929 11:21:25.276859 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20464,"spinlock_wait_cycles":1408,"thread_start_us":19565,"threads_started":1}
I20260929 11:21:25.277521 8227 leader_election.cc:302] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, 691fe906ef774e7b87a73ff24ec3e58b; no voters:
I20260929 11:21:25.277875 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce8d42438f04a47b4129c017d6d135a" candidate_uuid: "691fe906ef774e7b87a73ff24ec3e58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36610f683fe44548b1c09af2c7e34fa" is_pre_election: true
I20260929 11:21:25.277937 8515 raft_consensus.cc:2781] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:25.278051 8515 raft_consensus.cc:482] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:25.278087 8433 raft_consensus.cc:2454] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 691fe906ef774e7b87a73ff24ec3e58b in term 0.
I20260929 11:21:25.278148 8515 raft_consensus.cc:3037] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:25.281289 8515 raft_consensus.cc:504] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.281873 8515 leader_election.cc:288] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [CANDIDATE]: Term 1 election: Requested vote from peers 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141), a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259)
I20260929 11:21:25.282646 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce8d42438f04a47b4129c017d6d135a" candidate_uuid: "691fe906ef774e7b87a73ff24ec3e58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403e3f833f7e4fb28ccdc432393b9d90"
I20260929 11:21:25.282857 8156 raft_consensus.cc:3037] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:25.285876 8156 raft_consensus.cc:2454] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691fe906ef774e7b87a73ff24ec3e58b in term 1.
I20260929 11:21:25.286440 8227 leader_election.cc:302] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403e3f833f7e4fb28ccdc432393b9d90, 691fe906ef774e7b87a73ff24ec3e58b; no voters:
I20260929 11:21:25.286782 8515 raft_consensus.cc:2781] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:25.287577 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce8d42438f04a47b4129c017d6d135a" candidate_uuid: "691fe906ef774e7b87a73ff24ec3e58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36610f683fe44548b1c09af2c7e34fa"
I20260929 11:21:25.287806 8433 raft_consensus.cc:3037] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:25.290759 8433 raft_consensus.cc:2454] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691fe906ef774e7b87a73ff24ec3e58b in term 1.
I20260929 11:21:25.304829 8515 raft_consensus.cc:686] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 1 LEADER]: Becoming Leader. State: Replica: 691fe906ef774e7b87a73ff24ec3e58b, State: Running, Role: LEADER
I20260929 11:21:25.305368 8515 consensus_queue.cc:237] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } }
I20260929 11:21:25.316143 7998 catalog_manager.cc:5495] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b reported cstate change: term changed from 0 to 1, leader changed from <none> to 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "691fe906ef774e7b87a73ff24ec3e58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691fe906ef774e7b87a73ff24ec3e58b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20260929 11:21:25.341128 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.335184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.97.2:44865)
0929 11:21:25.335638 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.335643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.335665 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:21:25.335792 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.335797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.335900 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.336077 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:25.336506 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.337336 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.338691 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.339134 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.339266 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.339741 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:25.340448 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:25.340627 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.340707 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.340777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":559,"threads_started":1}
I20260929 11:21:25.358170 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.350518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36379 (local address 127.0.97.3:41259)
0929 11:21:25.350747 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.350756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.353270 (+ 2514us) server_negotiation.cc:407] Connection header received
0929 11:21:25.353536 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.353540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.353643 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.353732 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:25.354203 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.355013 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.356328 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.356874 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.357012 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.357402 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:25.357529 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:25.357640 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.357771 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.357842 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:21:25.358264 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.350396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.97.2:36379)
0929 11:21:25.353089 (+ 2693us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:25.353111 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:25.353374 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:25.353748 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:25.353752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:25.353764 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:21:25.354038 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.354046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.355150 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.355156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.356196 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.356206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.357041 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.357047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.357184 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.357233 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:21:25.357645 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:21:25.357657 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 11:21:25.357752 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 11:21:25.357874 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2552,"thread_start_us":2404,"threads_started":1}
I20260929 11:21:25.369510 8433 raft_consensus.cc:1261] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Refusing update from remote peer 691fe906ef774e7b87a73ff24ec3e58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:25.369529 8156 raft_consensus.cc:1261] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Refusing update from remote peer 691fe906ef774e7b87a73ff24ec3e58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:25.370283 8591 consensus_queue.cc:1035] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "403e3f833f7e4fb28ccdc432393b9d90" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:25.370704 8515 consensus_queue.cc:1035] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36610f683fe44548b1c09af2c7e34fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:25.426877 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.412336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53481 (local address 127.0.97.2:44865)
0929 11:21:25.412483 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.412489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.421337 (+ 8848us) server_negotiation.cc:407] Connection header received
0929 11:21:25.421629 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.421635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.421716 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.421780 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:25.422701 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.423475 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.424881 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.425358 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.425520 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.426106 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:25.426233 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:25.426298 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.426506 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.426578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 11:21:25.427623 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.412009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44865 (local address 127.0.97.3:53481)
0929 11:21:25.421245 (+ 9236us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:25.421267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:25.421453 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:25.421992 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:25.421996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:25.422008 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:21:25.422524 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.422538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.423624 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.423627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.424747 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:25.424758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.425663 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.425672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:25.425867 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.425914 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:21:25.426377 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:21:25.426387 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:21:25.427124 (+ 737us) client_negotiation.cc:240] Negotiation successful
0929 11:21:25.427250 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8934,"thread_start_us":8803,"threads_started":1}
I20260929 11:21:25.484231 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:25.484505 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:25.487920 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cf2bac47874eefbf0f2c19667c44ac: Sending DeleteTablet for 3 replicas of tablet fce8d42438f04a47b4129c017d6d135a
I20260929 11:21:25.488806 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet fce8d42438f04a47b4129c017d6d135a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54476
I20260929 11:21:25.489095 8532 tablet_replica.cc:330] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: stopping tablet replica
I20260929 11:21:25.489101 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet fce8d42438f04a47b4129c017d6d135a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35342
I20260929 11:21:25.489289 8532 raft_consensus.cc:2229] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:25.489434 8531 tablet_replica.cc:330] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: stopping tablet replica
I20260929 11:21:25.489499 8532 raft_consensus.cc:2258] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:25.489627 8531 raft_consensus.cc:2229] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:25.489806 8531 raft_consensus.cc:2258] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:25.490468 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet fce8d42438f04a47b4129c017d6d135a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40648
I20260929 11:21:25.490746 8533 tablet_replica.cc:330] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: stopping tablet replica
I20260929 11:21:25.490928 8533 raft_consensus.cc:2229] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:25.490964 8532 ts_tablet_manager.cc:1905] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:25.491415 8533 raft_consensus.cc:2258] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:25.492470 8531 ts_tablet_manager.cc:1905] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:25.497527 8533 ts_tablet_manager.cc:1905] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:25.498721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 8069
I20260929 11:21:25.504724 8533 ts_tablet_manager.cc:1918] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:21:25.504868 8533 log.cc:1196] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/fce8d42438f04a47b4129c017d6d135a
I20260929 11:21:25.505262 8533 ts_tablet_manager.cc:1939] T fce8d42438f04a47b4129c017d6d135a P 691fe906ef774e7b87a73ff24ec3e58b: Deleting consensus metadata
I20260929 11:21:25.506323 7982 catalog_manager.cc:4841] TS 691fe906ef774e7b87a73ff24ec3e58b (127.0.97.2:44865): tablet fce8d42438f04a47b4129c017d6d135a (table test-table [id=5bae06f9997245c1b93f3f09a067144b]) successfully deleted
I20260929 11:21:25.507747 8532 ts_tablet_manager.cc:1918] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:21:25.508358 8532 log.cc:1196] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/fce8d42438f04a47b4129c017d6d135a
I20260929 11:21:25.508983 8531 ts_tablet_manager.cc:1918] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:21:25.509131 8531 log.cc:1196] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/fce8d42438f04a47b4129c017d6d135a
I20260929 11:21:25.509429 8532 ts_tablet_manager.cc:1939] T fce8d42438f04a47b4129c017d6d135a P 403e3f833f7e4fb28ccdc432393b9d90: Deleting consensus metadata
I20260929 11:21:25.509631 8531 ts_tablet_manager.cc:1939] T fce8d42438f04a47b4129c017d6d135a P a36610f683fe44548b1c09af2c7e34fa: Deleting consensus metadata
I20260929 11:21:25.510679 7982 catalog_manager.cc:4841] TS a36610f683fe44548b1c09af2c7e34fa (127.0.97.3:41259): tablet fce8d42438f04a47b4129c017d6d135a (table test-table [id=5bae06f9997245c1b93f3f09a067144b]) successfully deleted
I20260929 11:21:25.510867 7984 catalog_manager.cc:4841] TS 403e3f833f7e4fb28ccdc432393b9d90 (127.0.97.1:34141): tablet fce8d42438f04a47b4129c017d6d135a (table test-table [id=5bae06f9997245c1b93f3f09a067144b]) successfully deleted
Sep 29 11:21:25 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 11:21:25.530751 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.499997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.0.97.1:34141)
0929 11:21:25.500154 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.500160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.513691 (+ 13531us) server_negotiation.cc:407] Connection header received
0929 11:21:25.513860 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.513866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.513965 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.514071 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:25.515515 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.516643 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.518386 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.519115 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.519237 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.524064 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:25.524074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:25.524092 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:25.524134 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:25.527660 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.528459 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.528468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.528472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.528649 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.529021 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.529027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.529031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.529438 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:25.529722 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.529993 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.530199 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:21:25.531472 8200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:25.650512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 8069
I20260929 11:21:25.680765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 8209
Sep 29 11:21:25 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 11:21:25.696513 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.681567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.0.97.2:44865)
0929 11:21:25.681778 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.681786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.681806 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:21:25.682009 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.682015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.682106 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.682223 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:25.683464 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.684204 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.685562 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.685999 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.686150 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.690193 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:25.690204 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:25.690225 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:25.690260 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:25.694028 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.694686 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.694695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.694698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.694831 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.695126 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.695130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.695134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.695523 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:25.695837 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.695974 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.696175 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:21:25.697261 8339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:25.818014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 8209
I20260929 11:21:25.837772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 8348
Sep 29 11:21:25 dist-test-slave-srst krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 11:21:25.852743 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:25.838619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.0.97.3:41259)
0929 11:21:25.838832 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:25.838839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:25.838864 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:21:25.838935 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:25.838939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:25.838997 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:25.839113 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:25.840417 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.841206 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.842512 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:25.842944 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:25.843102 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:25.847340 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:25.847355 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:25.847371 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:25.847402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:25.850095 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.850703 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.850709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.850712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.850826 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:25.851178 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:25.851184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:25.851187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:25.851545 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:25.851848 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:25.852123 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 11:21:25.852347 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:21:25.853513 8478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:25.993847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 8348
I20260929 11:21:26.021914 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 7966
I20260929 11:21:26.023012 8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:26.135493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 7966
2026-09-29T11:21:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5729 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:26 dist-test-slave-srst krb5kdc[8611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:26 dist-test-slave-srst krb5kdc[8611](info): set up 2 sockets
Sep 29 11:21:26 dist-test-slave-srst krb5kdc[8611](info): commencing operation
krb5kdc: starting...
W20260929 11:21:27.209726 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:27 dist-test-slave-srst krb5kdc[8611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:21:27.419974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40773
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:27.744036 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:27.744344 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:27.744441 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:27.751829 8624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:27.751961 8624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:27.752015 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:27.752105 8624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:27.752177 8624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:27.762373 8624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:27.763756 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:27.765127 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:27.776847 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:27.776885 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:27.777285 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:27.777555 8624 server_base.cc:1034] running on GCE node
I20260929 11:21:27.778239 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:27.779626 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:27.780815 8624 hybrid_clock.cc:648] HybridClock initialized: now 1790680887780774 us; error 33 us; skew 500 ppm
I20260929 11:21:27.781152 8624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T11:21:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1725 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:27 dist-test-slave-srst krb5kdc[8638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:27 dist-test-slave-srst krb5kdc[8638](info): set up 2 sockets
Sep 29 11:21:27 dist-test-slave-srst krb5kdc[8638](info): commencing operation
krb5kdc: starting...
W20260929 11:21:28.937495 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:29 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:21:29.146075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:29.457494 8651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:29.457788 8651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:29.457971 8651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:29.465041 8651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:29.465132 8651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:29.465240 8651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:29.465350 8651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:29.465426 8651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:29.475296 8651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:29.476539 8651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:29.477811 8651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:29.489110 8656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:29.489046 8657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:29.489066 8659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:29.490134 8651 server_base.cc:1034] running on GCE node
I20260929 11:21:29.491117 8651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:29.492940 8651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:29.494184 8651 hybrid_clock.cc:648] HybridClock initialized: now 1790680889494125 us; error 50 us; skew 500 ppm
I20260929 11:21:29.494542 8651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:29 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:29.497578 8651 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:21:29.498984 8651 webserver.cc:460] Webserver started at http://127.0.97.62:33841/ using document root <none> and password file <none>
I20260929 11:21:29.499672 8651 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:29.499812 8651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:29.500121 8651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:29.503037 8651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "273df01a28d840ed894fe72af425ad4d"
format_stamp: "Formatted at 2026-09-29 11:21:29 on dist-test-slave-srst"
server_key: "b02c37540a78cc1f8ba3679a9579e9d8"
server_key_iv: "3bc2405b5a5c1f0a170f3218f3e818a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:29.503810 8651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "273df01a28d840ed894fe72af425ad4d"
format_stamp: "Formatted at 2026-09-29 11:21:29 on dist-test-slave-srst"
server_key: "b02c37540a78cc1f8ba3679a9579e9d8"
server_key_iv: "3bc2405b5a5c1f0a170f3218f3e818a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:29.508793 8651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 11:21:29.511981 8666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:29.513257 8651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:29.513437 8651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "273df01a28d840ed894fe72af425ad4d"
format_stamp: "Formatted at 2026-09-29 11:21:29 on dist-test-slave-srst"
server_key: "b02c37540a78cc1f8ba3679a9579e9d8"
server_key_iv: "3bc2405b5a5c1f0a170f3218f3e818a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:29.513638 8651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:29.534438 8651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:29.538496 8651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:29.538781 8651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:29.557098 8651 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44829
I20260929 11:21:29.557130 8717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44829 every 8 connection(s)
I20260929 11:21:29.558468 8651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:29.562217 8718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:29.566190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 8651
I20260929 11:21:29.566387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:29.567009 388 external_mini_cluster.cc:1428] Setting key 9a061d7e2052e635a1894db0bf53c3f2
I20260929 11:21:29.572947 8718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Bootstrap starting.
I20260929 11:21:29.576532 8718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:29.577708 8718 log.cc:824] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:29.580332 8718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: No bootstrap required, opened a new log
Sep 29 11:21:29 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:29.590231 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:29.568936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.0.97.62:44829)
0929 11:21:29.569786 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:29.569796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:29.569842 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:21:29.571506 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:29.571535 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:29.571929 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:29.572257 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:29.574102 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:29.575412 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:29.577244 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:29.577943 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:29.578073 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:29.581977 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:29.581987 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:29.582004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:29.582061 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:29.585960 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:29.586613 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:29.586621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:29.586635 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:29.586829 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:29.587164 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:29.587169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:29.587171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:29.587816 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:29.588117 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:29.588396 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 11:21:29.588848 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":342,"threads_started":1}
I20260929 11:21:29.591328 8718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:29.591779 8718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:29.591897 8718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273df01a28d840ed894fe72af425ad4d, State: Initialized, Role: FOLLOWER
I20260929 11:21:29.592458 8718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:29.592769 8718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:29.592921 8718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:29.593113 8718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:29.595225 8718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:29.595705 8718 leader_election.cc:302] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273df01a28d840ed894fe72af425ad4d; no voters:
I20260929 11:21:29.596259 8718 leader_election.cc:288] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:29.596378 8723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:29.597184 8723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 LEADER]: Becoming Leader. State: Replica: 273df01a28d840ed894fe72af425ad4d, State: Running, Role: LEADER
I20260929 11:21:29.597813 8723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:29.598377 8718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:29.600730 8724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273df01a28d840ed894fe72af425ad4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } } }
I20260929 11:21:29.600867 8725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273df01a28d840ed894fe72af425ad4d. Latest consensus state: current_term: 1 leader_uuid: "273df01a28d840ed894fe72af425ad4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } } }
I20260929 11:21:29.601187 8724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:29.601199 8725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:29.610030 8732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:29.614029 8732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:29.622172 8732 catalog_manager.cc:1347] Generated new cluster ID: b086b9f5911a4e9fbe8d95a62382634c
I20260929 11:21:29.622273 8732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:29.640286 8732 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:29.642284 8732 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:29.659194 8732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Generated new TSK 0
I20260929 11:21:29.660065 8732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:21:29.722203 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44829
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:30.042092 8744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:30.042467 8744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:30.042642 8744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:30.049664 8744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:30.049816 8744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:30.050011 8744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:30.059226 8744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:30.060489 8744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:30.061770 8744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:30.074600 8750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:30.074821 8749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:30.074585 8752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:30.075366 8744 server_base.cc:1034] running on GCE node
I20260929 11:21:30.075917 8744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:30.076874 8744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:30.078078 8744 hybrid_clock.cc:648] HybridClock initialized: now 1790680890078041 us; error 32 us; skew 500 ppm
I20260929 11:21:30.078366 8744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:30 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:30.081465 8744 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:21:30.082805 8744 webserver.cc:460] Webserver started at http://127.0.97.1:41183/ using document root <none> and password file <none>
I20260929 11:21:30.083462 8744 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:30.083554 8744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:30.083793 8744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:30.086453 8744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "0d94479f9e614f0a9d47017b68fcca88"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "2992d716133dcb2b0ecae7d752bfe6ff"
server_key_iv: "55bf57651852c7986df0641af34beb3a"
server_key_version: "encryptionkey@0"
I20260929 11:21:30.087182 8744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0d94479f9e614f0a9d47017b68fcca88"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "2992d716133dcb2b0ecae7d752bfe6ff"
server_key_iv: "55bf57651852c7986df0641af34beb3a"
server_key_version: "encryptionkey@0"
I20260929 11:21:30.092020 8744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:21:30.095255 8759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:30.096436 8744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 11:21:30.096649 8744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "0d94479f9e614f0a9d47017b68fcca88"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "2992d716133dcb2b0ecae7d752bfe6ff"
server_key_iv: "55bf57651852c7986df0641af34beb3a"
server_key_version: "encryptionkey@0"
I20260929 11:21:30.096824 8744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:30.122380 8744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:30.126613 8744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:30.126950 8744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:30.127864 8744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:30.129267 8744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:30.129346 8744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:30.129519 8744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:30.129601 8744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:30.163516 8744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46061
I20260929 11:21:30.163533 8871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46061 every 8 connection(s)
I20260929 11:21:30.164948 8744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:30.174598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 8744
I20260929 11:21:30.174794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:30.175402 388 external_mini_cluster.cc:1428] Setting key 03b8fd3c3917e10124e0cdfd7895ccd5
Sep 29 11:21:30 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:30.184443 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:30.167353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49939 (local address 127.0.97.62:44829)
0929 11:21:30.167843 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:30.167847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:30.168745 (+ 898us) server_negotiation.cc:407] Connection header received
0929 11:21:30.170556 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:30.170561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:30.170650 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:30.170757 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:30.172131 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.172883 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.174436 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.174893 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.175047 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:30.178212 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:30.178220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:30.178228 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:30.178268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:30.181655 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:30.182280 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:30.182287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:30.182293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:30.182418 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:30.182762 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:30.182769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:30.182775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:30.183126 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:30.183261 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:30.183845 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 11:21:30.184095 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":286,"threads_started":1}
I20260929 11:21:30.185527 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:30.167824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:49939)
0929 11:21:30.168513 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:30.168555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:30.170291 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:30.170970 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:30.170978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:30.171437 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:30.171973 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:30.171987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.173037 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.173042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:30.174287 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:30.174298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.175149 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.175156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:30.175355 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:30.176007 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:30.176026 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:30.177999 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:30.181787 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:30.181793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:30.181804 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:30.182165 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:30.182513 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:30.182516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:30.182518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:30.182666 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:30.183240 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:30.183244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:30.183514 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 11:21:30.183718 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 11:21:30.184081 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":282,"threads_started":1}
I20260929 11:21:30.187005 8872 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44829
I20260929 11:21:30.187318 8872 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:30.188205 8872 heartbeater.cc:511] Master 127.0.97.62:44829 requested a full tablet report, sending...
I20260929 11:21:30.190404 8683 ts_manager.cc:194] Registered new tserver with Master: 0d94479f9e614f0a9d47017b68fcca88 (127.0.97.1:46061)
I20260929 11:21:30.191814 8683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49939
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:21:30.221918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44829
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:30.533592 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:30.533982 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:30.534152 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:30.541115 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:30.541255 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:30.541435 8878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:30.550789 8878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:30.552090 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:30.553392 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:30.567370 8886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:30.567397 8884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:30.568519 8878 server_base.cc:1034] running on GCE node
W20260929 11:21:30.572837 8883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:30.573558 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:30.574478 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:30.575690 8878 hybrid_clock.cc:648] HybridClock initialized: now 1790680890575620 us; error 56 us; skew 500 ppm
I20260929 11:21:30.576116 8878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:30 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:30.579145 8878 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:21:30.580544 8878 webserver.cc:460] Webserver started at http://127.0.97.2:43901/ using document root <none> and password file <none>
I20260929 11:21:30.581229 8878 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:30.581393 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:30.581739 8878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:30.584697 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "fffb093b252b4d64a34b3f7a7938b86a"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "6c3dede9b7160112800134a577a9c871"
server_key_iv: "803c91f098c89d8c842f3075de0a6517"
server_key_version: "encryptionkey@0"
I20260929 11:21:30.585435 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fffb093b252b4d64a34b3f7a7938b86a"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "6c3dede9b7160112800134a577a9c871"
server_key_iv: "803c91f098c89d8c842f3075de0a6517"
server_key_version: "encryptionkey@0"
I20260929 11:21:30.590315 8878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:21:30.593389 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:30.594601 8878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 11:21:30.594780 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "fffb093b252b4d64a34b3f7a7938b86a"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "6c3dede9b7160112800134a577a9c871"
server_key_iv: "803c91f098c89d8c842f3075de0a6517"
server_key_version: "encryptionkey@0"
I20260929 11:21:30.594976 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:30.627663 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:30.631561 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:30.631844 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:30.632794 8878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:30.634104 8878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:30.634232 8878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:30.634383 8878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:30.634455 8878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:30.666138 8878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46679
I20260929 11:21:30.666184 9005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46679 every 8 connection(s)
I20260929 11:21:30.667487 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:30.671260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 8878
I20260929 11:21:30.671432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:30.671970 388 external_mini_cluster.cc:1428] Setting key 4617c7c39d3c2b38aa2b1e8f5d83e25b
Sep 29 11:21:30 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:30.688432 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:30.669997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46929 (local address 127.0.97.62:44829)
0929 11:21:30.670172 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:30.670178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:30.671593 (+ 1415us) server_negotiation.cc:407] Connection header received
0929 11:21:30.673915 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:30.673920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:30.674000 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:30.674077 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:30.675553 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.676733 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.678263 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.679000 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.679135 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:30.682626 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:30.682630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:30.682634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:30.682662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:30.685096 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:30.685892 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:30.685900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:30.685907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:30.686059 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:30.686535 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:30.686542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:30.686548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:30.686924 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:30.687074 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:30.687707 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 11:21:30.687979 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:21:30.689044 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:30.670521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:46929)
0929 11:21:30.671409 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:30.671468 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:30.673591 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:30.674333 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:30.674341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:30.674831 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:30.675385 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:30.675402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.676868 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.676872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:30.678114 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:30.678125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:30.679193 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:30.679216 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:30.679401 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:30.680059 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:30.680084 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:30.682422 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:30.685225 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:30.685234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:30.685249 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:30.685753 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:30.686168 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:30.686171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:30.686173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:30.686409 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:30.687042 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:30.687048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:30.687345 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 11:21:30.687563 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 11:21:30.687942 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":343,"threads_started":1}
I20260929 11:21:30.690436 9006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44829
I20260929 11:21:30.690738 9006 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:30.691430 9006 heartbeater.cc:511] Master 127.0.97.62:44829 requested a full tablet report, sending...
I20260929 11:21:30.692874 8683 ts_manager.cc:194] Registered new tserver with Master: fffb093b252b4d64a34b3f7a7938b86a (127.0.97.2:46679)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 11:21:30.693604 8683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:21:30.720019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44829
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:31.036064 9011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:31.036401 9011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:31.036576 9011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:31.043756 9011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:31.043915 9011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:31.044103 9011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:31.053791 9011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:31.055104 9011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:31.056437 9011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:31.069607 9019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:31.069561 9017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:31.069641 9016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:31.069958 9011 server_base.cc:1034] running on GCE node
I20260929 11:21:31.070945 9011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:31.071813 9011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:31.073006 9011 hybrid_clock.cc:648] HybridClock initialized: now 1790680891072954 us; error 39 us; skew 500 ppm
I20260929 11:21:31.073333 9011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:31 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:31.076370 9011 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:21:31.077896 9011 webserver.cc:460] Webserver started at http://127.0.97.3:39281/ using document root <none> and password file <none>
I20260929 11:21:31.078579 9011 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:31.078723 9011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:31.079030 9011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:31.081918 9011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "3c531aa52f5047f1a970930b07c26828"
format_stamp: "Formatted at 2026-09-29 11:21:31 on dist-test-slave-srst"
server_key: "9ed31c9202c717e33d21312b799c8026"
server_key_iv: "ea0ca536d70370ab2bd97e6c95dd34cd"
server_key_version: "encryptionkey@0"
I20260929 11:21:31.082705 9011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3c531aa52f5047f1a970930b07c26828"
format_stamp: "Formatted at 2026-09-29 11:21:31 on dist-test-slave-srst"
server_key: "9ed31c9202c717e33d21312b799c8026"
server_key_iv: "ea0ca536d70370ab2bd97e6c95dd34cd"
server_key_version: "encryptionkey@0"
I20260929 11:21:31.087837 9011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:21:31.091207 9026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:31.092494 9011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:21:31.092693 9011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "3c531aa52f5047f1a970930b07c26828"
format_stamp: "Formatted at 2026-09-29 11:21:31 on dist-test-slave-srst"
server_key: "9ed31c9202c717e33d21312b799c8026"
server_key_iv: "ea0ca536d70370ab2bd97e6c95dd34cd"
server_key_version: "encryptionkey@0"
I20260929 11:21:31.092916 9011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:31.127815 9011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:31.132184 9011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:31.132474 9011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:31.133425 9011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:31.134738 9011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:31.134848 9011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:31.135001 9011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:31.135074 9011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:31.166927 9011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46301
I20260929 11:21:31.166977 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46301 every 8 connection(s)
I20260929 11:21:31.168154 9011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:31.168553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9011
I20260929 11:21:31.168788 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:31.169404 388 external_mini_cluster.cc:1428] Setting key b4f936b828ed3dc9170b1b0153b6aa0c
Sep 29 11:21:31 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:31.187225 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:31.170827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37773 (local address 127.0.97.62:44829)
0929 11:21:31.171016 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:31.171020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:31.172381 (+ 1361us) server_negotiation.cc:407] Connection header received
0929 11:21:31.174203 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:31.174207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:31.174268 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:31.174350 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:31.175652 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.176379 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.177861 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.178244 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.178332 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:31.181515 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:31.181520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:31.181523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:31.181551 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:31.184269 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.185052 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.185057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.185062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.185203 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.185618 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.185628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.185630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.185874 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:31.186024 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:31.186682 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 11:21:31.186885 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:21:31.188170 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:31.171445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:37773)
0929 11:21:31.172234 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:31.172286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:31.173971 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:31.174508 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:31.174516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:31.174990 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:31.175518 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:31.175534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.176498 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.176501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:31.177748 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:31.177759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.178423 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.178428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:31.178617 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:31.179217 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:31.179239 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:31.181344 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:31.184408 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:31.184416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:31.184432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:31.184919 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:31.185310 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:31.185316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:31.185319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:31.185500 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:31.185999 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:31.186007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:31.186322 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 11:21:31.186571 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 11:21:31.187056 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260929 11:21:31.189390 9139 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44829
I20260929 11:21:31.189682 9139 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:31.190431 9139 heartbeater.cc:511] Master 127.0.97.62:44829 requested a full tablet report, sending...
I20260929 11:21:31.191637 8683 ts_manager.cc:194] Registered new tserver with Master: 3c531aa52f5047f1a970930b07c26828 (127.0.97.3:46301)
I20260929 11:21:31.192359 8683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37773
I20260929 11:21:31.194859 8872 heartbeater.cc:503] Master 127.0.97.62:44829 was elected leader, sending a full tablet report...
I20260929 11:21:31.196874 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 11:21:31.325538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 8744
Sep 29 11:21:31 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:21:31.341497 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:31.326757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.0.97.1:46061)
0929 11:21:31.327407 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:31.327413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:31.327441 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:21:31.327500 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:31.327506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:31.327709 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:31.327854 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:31.329137 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.330147 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.331335 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.331862 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.331953 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:31.335825 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:31.335844 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:31.335859 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:31.335895 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:31.338723 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.339439 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.339447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.339449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.339572 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.339925 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.339932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.339934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.340256 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:31.340548 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:31.340878 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 11:21:31.341144 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":355,"threads_started":1}
I20260929 11:21:31.342757 8867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:31.423810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 8744
I20260929 11:21:31.440006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 8878
Sep 29 11:21:31 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:21:31.455391 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:31.441057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.97.2:46679)
0929 11:21:31.441703 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:31.441710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:31.441740 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:21:31.441805 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:31.441812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:31.442023 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:31.442162 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:31.443323 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.444354 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.445586 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.446131 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.446230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:31.450198 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:31.450209 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:31.450222 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:31.450254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:31.453027 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.453658 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.453664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.453667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.453777 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.454140 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.454143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.454144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.454451 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:31.454645 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:31.454896 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 11:21:31.455077 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
I20260929 11:21:31.456579 9001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:31.536899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 8878
I20260929 11:21:31.553234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9011
Sep 29 11:21:31 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:21:31.571077 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:31.554283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.0.97.3:46301)
0929 11:21:31.554909 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:31.554917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:31.554946 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:21:31.555003 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:31.555009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:31.555220 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:31.555387 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:31.556742 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.557841 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.559685 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:31.560215 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:31.560372 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:31.565301 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:31.565312 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:31.565327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:31.565361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:31.568216 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.569089 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.569096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.569098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.569224 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:31.569609 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:31.569613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:31.569616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:31.569937 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:31.570256 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:31.570501 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 11:21:31.570756 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":317,"threads_started":1}
I20260929 11:21:31.572358 9134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:31.652865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9011
I20260929 11:21:31.668433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 8651
I20260929 11:21:31.669668 8713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:31.769661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 8651
I20260929 11:21:31.789562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44829
--webserver_interface=127.0.97.62
--webserver_port=33841
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:32.106840 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:32.107225 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:32.107402 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:32.114473 9156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:32.114619 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:32.114727 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:32.114830 9156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:32.114910 9156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:32.124506 9156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:32.125945 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:32.127302 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:32.139263 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:32.139222 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:32.139207 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:32.140064 9156 server_base.cc:1034] running on GCE node
I20260929 11:21:32.141012 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:32.142388 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:32.143623 9156 hybrid_clock.cc:648] HybridClock initialized: now 1790680892143581 us; error 57 us; skew 500 ppm
I20260929 11:21:32.143973 9156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:32.147153 9156 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:21:32.148748 9156 webserver.cc:460] Webserver started at http://127.0.97.62:33841/ using document root <none> and password file <none>
I20260929 11:21:32.149433 9156 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:32.149590 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:32.154966 9156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 11:21:32.157788 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:32.159073 9156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:32.159260 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "273df01a28d840ed894fe72af425ad4d"
format_stamp: "Formatted at 2026-09-29 11:21:29 on dist-test-slave-srst"
server_key: "b02c37540a78cc1f8ba3679a9579e9d8"
server_key_iv: "3bc2405b5a5c1f0a170f3218f3e818a5"
server_key_version: "encryptionkey@0"
I20260929 11:21:32.160199 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:32.197505 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:32.202057 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:32.202350 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:32.221343 9156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44829
I20260929 11:21:32.221398 9222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44829 every 8 connection(s)
I20260929 11:21:32.222851 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:32.229779 9223 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:21:32.230855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9156
I20260929 11:21:32.231803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46061
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44829
--webserver_port=41183
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:21:32.235028 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Bootstrap starting.
I20260929 11:21:32.255293 9223 log.cc:824] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:32.264842 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:21:32.265393 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Bootstrap complete.
I20260929 11:21:32.274804 9223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:32.275619 9223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273df01a28d840ed894fe72af425ad4d, State: Initialized, Role: FOLLOWER
I20260929 11:21:32.276248 9223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:32.276489 9223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:32.276621 9223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:32.276813 9223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:21:32.280637 9223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:32.281147 9223 leader_election.cc:302] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273df01a28d840ed894fe72af425ad4d; no voters:
I20260929 11:21:32.281772 9223 leader_election.cc:288] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:21:32.281909 9227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:21:32.283308 9227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [term 2 LEADER]: Becoming Leader. State: Replica: 273df01a28d840ed894fe72af425ad4d, State: Running, Role: LEADER
I20260929 11:21:32.283921 9227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } }
I20260929 11:21:32.284102 9223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:32.286671 9229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273df01a28d840ed894fe72af425ad4d. Latest consensus state: current_term: 2 leader_uuid: "273df01a28d840ed894fe72af425ad4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } } }
I20260929 11:21:32.286705 9228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "273df01a28d840ed894fe72af425ad4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273df01a28d840ed894fe72af425ad4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44829 } } }
I20260929 11:21:32.287482 9228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:32.288277 9229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:32.296877 9233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:32.303864 9233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:32.305388 9233 catalog_manager.cc:1259] Loaded cluster ID: b086b9f5911a4e9fbe8d95a62382634c
I20260929 11:21:32.305563 9233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:32.309650 9233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:32.311162 9233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 273df01a28d840ed894fe72af425ad4d: Loaded TSK: 0
I20260929 11:21:32.312796 9233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:21:32.590461 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:32.590765 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:32.590934 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:32.598819 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:32.598943 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:32.599123 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:32.610000 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41183
--tserver_master_addrs=127.0.97.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:32.611397 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:32.612870 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:32.626884 9252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:32.627115 9250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:32.627158 9249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:32.627914 9225 server_base.cc:1034] running on GCE node
I20260929 11:21:32.628708 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:32.629917 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:32.631222 9225 hybrid_clock.cc:648] HybridClock initialized: now 1790680892631161 us; error 62 us; skew 500 ppm
I20260929 11:21:32.631608 9225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:32.635656 9225 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 11:21:32.637347 9225 webserver.cc:460] Webserver started at http://127.0.97.1:41183/ using document root <none> and password file <none>
I20260929 11:21:32.638087 9225 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:32.638265 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:32.644061 9225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 11:21:32.647204 9259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:32.648545 9225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:21:32.648759 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "0d94479f9e614f0a9d47017b68fcca88"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "2992d716133dcb2b0ecae7d752bfe6ff"
server_key_iv: "55bf57651852c7986df0641af34beb3a"
server_key_version: "encryptionkey@0"
I20260929 11:21:32.649744 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:32.695012 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:32.699373 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:32.699678 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:32.700736 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:32.702013 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:32.702167 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:32.702306 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:32.702395 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:32.735380 9225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46061
I20260929 11:21:32.735471 9371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46061 every 8 connection(s)
I20260929 11:21:32.736971 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:32.747545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9225
I20260929 11:21:32.748260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46679
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44829
--webserver_port=43901
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:32.756743 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:32.740022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:59679)
0929 11:21:32.741378 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:32.741431 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:32.743011 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:32.744297 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:32.744304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:32.744876 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:32.745415 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:32.745431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.746916 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.746920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:32.749018 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:32.749036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.750015 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.750024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:32.750337 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:32.751045 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:32.751088 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:32.755729 (+ 4641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":561,"threads_started":1}
W20260929 11:21:32.757390 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:32.740003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59679 (local address 127.0.97.62:44829)
0929 11:21:32.741637 (+ 1634us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:32.741647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:32.741687 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:21:32.743516 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:32.743530 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:32.743870 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:32.744168 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:32.745576 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.746782 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.749166 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.749792 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.749916 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:32.756139 (+ 6223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59679: BlockingRecv error: recv got EOF from 127.0.97.1:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":1202,"threads_started":1}
W20260929 11:21:32.757618 9372 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44829: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:32.766896 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:32.758202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:51387)
0929 11:21:32.758368 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:32.758387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:32.758483 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:32.758833 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:32.758836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:32.759121 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:32.759261 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:32.759268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.760730 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.760734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:32.761616 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:32.761626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.762607 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.762611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:32.762727 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:32.763301 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:32.763317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:32.766701 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:21:32.767257 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:32.758343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51387 (local address 127.0.97.62:44829)
0929 11:21:32.758528 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:32.758542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:32.758561 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:21:32.758618 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:32.758626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:32.758704 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:32.758808 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:32.759396 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.760571 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.761750 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.762465 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.762610 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:32.766976 (+ 4366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51387: BlockingRecv error: recv got EOF from 127.0.97.1:51387 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:32 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:32.776262 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:32.767760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:55815)
0929 11:21:32.767921 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:32.767945 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:32.768019 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:32.768460 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:32.768464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:32.768809 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:32.769019 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:32.769033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.770578 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.770581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:32.771462 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:32.771472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.772489 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.772493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:32.772677 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:32.773272 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:32.773288 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:32.775985 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:21:32.776319 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:32.767920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55815 (local address 127.0.97.62:44829)
0929 11:21:32.768100 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:32.768107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:32.768129 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:21:32.768186 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:32.768194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:32.768288 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:32.768404 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:32.769166 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.770429 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.771615 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:32.772354 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:32.772502 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:32.776149 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55815: BlockingRecv error: recv got EOF from 127.0.97.1:55815 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 11:21:32.776803 9372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:21:33.098181 9376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:33.098503 9376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:33.098682 9376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:33.105909 9376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:33.106065 9376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:33.106264 9376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:33.115976 9376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43901
--tserver_master_addrs=127.0.97.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:33.117375 9376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:33.118721 9376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:33.132067 9381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:33.132073 9382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:33.132086 9384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:33.132725 9376 server_base.cc:1034] running on GCE node
I20260929 11:21:33.133374 9376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:33.134315 9376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:33.135517 9376 hybrid_clock.cc:648] HybridClock initialized: now 1790680893135472 us; error 39 us; skew 500 ppm
I20260929 11:21:33.135936 9376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:33.138988 9376 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 11:21:33.140501 9376 webserver.cc:460] Webserver started at http://127.0.97.2:43901/ using document root <none> and password file <none>
I20260929 11:21:33.141204 9376 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:33.141343 9376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:33.146620 9376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 11:21:33.149339 9391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:33.150770 9376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:33.150935 9376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "fffb093b252b4d64a34b3f7a7938b86a"
format_stamp: "Formatted at 2026-09-29 11:21:30 on dist-test-slave-srst"
server_key: "6c3dede9b7160112800134a577a9c871"
server_key_iv: "803c91f098c89d8c842f3075de0a6517"
server_key_version: "encryptionkey@0"
I20260929 11:21:33.151902 9376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:33.168720 9376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:33.173288 9376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:33.173529 9376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:33.174418 9376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:33.175663 9376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:33.175746 9376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:33.175829 9376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:33.175885 9376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:33.206306 9376 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46679
I20260929 11:21:33.206354 9503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46679 every 8 connection(s)
I20260929 11:21:33.207855 9376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:33.214195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9376
I20260929 11:21:33.214931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46301
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44829
--webserver_port=39281
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:21:33.226311 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.210392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59905 (local address 127.0.97.62:44829)
0929 11:21:33.210605 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.210611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.211841 (+ 1230us) server_negotiation.cc:407] Connection header received
0929 11:21:33.213661 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.213666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.213732 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.213818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.215924 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.216676 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.219145 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.219529 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.219629 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.226082 (+ 6453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59905: BlockingRecv error: recv got EOF from 127.0.97.2:59905 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:21:33.227330 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.210974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:59905)
0929 11:21:33.211673 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.211719 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.213409 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.214127 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.214141 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.214877 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.215735 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.215764 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.216828 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.216836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.218999 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.219024 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.219744 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.219753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.220095 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.221229 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.221251 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.225884 (+ 4633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":272,"threads_started":1}
W20260929 11:21:33.228047 9504 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44829: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:33.237593 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.228655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:54243)
0929 11:21:33.228823 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.228842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.228937 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.229302 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.229307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.229679 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.229870 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.229880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.231424 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.231429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.232691 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.232702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.233761 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.233765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.233879 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.234486 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.234503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.237402 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 11:21:33.238121 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.228728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54243 (local address 127.0.97.62:44829)
0929 11:21:33.228875 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.228882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.228899 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:21:33.229076 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.229085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.229168 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.229262 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.230019 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.231275 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.232819 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.233613 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.233771 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.237681 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54243: BlockingRecv error: recv got EOF from 127.0.97.2:54243 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:33.246552 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.238269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:55691)
0929 11:21:33.238492 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.238510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.238610 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.238965 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.238969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.239214 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.239341 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.239348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.240830 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.240833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.241737 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.241746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.242612 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.242616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.242727 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.243312 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.243328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.246270 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 11:21:33.246588 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.238347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55691 (local address 127.0.97.62:44829)
0929 11:21:33.238541 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.238548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.238570 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:21:33.238735 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.238743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.238828 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.238934 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.239472 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.240706 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.241857 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.242472 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.242590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.246426 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55691: BlockingRecv error: recv got EOF from 127.0.97.2:55691 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 11:21:33.246951 9504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:21:33.544340 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:33.544948 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:33.545056 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:33.552557 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:33.552727 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:33.552842 9507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:33.562827 9507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39281
--tserver_master_addrs=127.0.97.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:33.564198 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:33.565527 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:33.577989 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:33.578071 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:33.578102 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:33.579321 9507 server_base.cc:1034] running on GCE node
I20260929 11:21:33.579954 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:33.580875 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:33.582072 9507 hybrid_clock.cc:648] HybridClock initialized: now 1790680893582014 us; error 62 us; skew 500 ppm
I20260929 11:21:33.582370 9507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:33.585485 9507 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 11:21:33.586764 9507 webserver.cc:460] Webserver started at http://127.0.97.3:39281/ using document root <none> and password file <none>
I20260929 11:21:33.587397 9507 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:33.587502 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:33.592710 9507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:21:33.595196 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:33.596415 9507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:21:33.596535 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "3c531aa52f5047f1a970930b07c26828"
format_stamp: "Formatted at 2026-09-29 11:21:31 on dist-test-slave-srst"
server_key: "9ed31c9202c717e33d21312b799c8026"
server_key_iv: "ea0ca536d70370ab2bd97e6c95dd34cd"
server_key_version: "encryptionkey@0"
I20260929 11:21:33.597451 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:33.619048 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:33.623026 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:33.623247 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:33.624137 9507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:33.625417 9507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:33.625506 9507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:33.625577 9507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:33.625633 9507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:33.659102 9507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46301
I20260929 11:21:33.659197 9634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46301 every 8 connection(s)
I20260929 11:21:33.660575 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:33.670249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9507
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:21:33.677223 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.663207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41745 (local address 127.0.97.62:44829)
0929 11:21:33.663372 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.663376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.664569 (+ 1193us) server_negotiation.cc:407] Connection header received
0929 11:21:33.666465 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.666471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.666593 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.666709 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.668301 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.669193 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.670859 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.671448 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.671608 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.676929 (+ 5321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41745: BlockingRecv error: recv got EOF from 127.0.97.3:41745 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:21:33.678120 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.663630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:41745)
0929 11:21:33.664418 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.664457 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.666165 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.667008 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.667017 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.667539 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.668126 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.668143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.669340 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.669370 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.670675 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.670694 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.671859 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.671866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.672163 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.672964 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.672992 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.676769 (+ 3777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":339,"threads_started":1}
W20260929 11:21:33.678951 9635 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44829: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:33.687077 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.671270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.97.62:44829)
0929 11:21:33.672081 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.672085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.672098 (+ 13us) server_negotiation.cc:407] Connection header received
0929 11:21:33.672140 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.672143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.672192 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.672270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.673914 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.675013 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.676359 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.676969 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.677094 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.680203 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:33.680211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:33.680226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:33.680255 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:33.683811 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:33.684660 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:33.684667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:33.684673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:33.684842 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:33.685220 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:33.685223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:33.685225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:33.685720 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:33.685997 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:33.686369 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 11:21:33.686684 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":611,"threads_started":1}
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:33.688197 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.679505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:41717)
0929 11:21:33.679669 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.679689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.679789 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.680065 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.680070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.680343 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.680467 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.680474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.681892 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.681896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.682936 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.682947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.683948 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.683953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.684147 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.684723 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.684740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.687948 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:21:33.688242 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.679656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41717 (local address 127.0.97.62:44829)
0929 11:21:33.679799 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.679802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.679817 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:21:33.679880 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.679883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.679929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.679997 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.680628 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.681743 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.683090 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.683792 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.683920 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.688085 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41717: BlockingRecv error: recv got EOF from 127.0.97.3:41717 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:33.696467 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.688981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:56593)
0929 11:21:33.689146 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.689165 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.689259 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.689531 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.689535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.689823 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.690024 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.690035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.691283 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.691287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.692161 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.692170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.692794 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.692798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.692910 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.693349 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.693364 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.696269 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 11:21:33.696681 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.689059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56593 (local address 127.0.97.62:44829)
0929 11:21:33.689280 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.689287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.689305 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:21:33.689350 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.689354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.689417 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.689510 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.690171 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.691173 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.692267 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.692686 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.692772 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.696423 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56593: BlockingRecv error: recv got EOF from 127.0.97.3:56593 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260929 11:21:33.696834 9635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:33 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:33.785588 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:33.777378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:51295)
0929 11:21:33.777913 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:33.777928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:33.778037 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:33.778303 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:33.778306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:33.778754 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:33.778941 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.778949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.779885 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.779889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.780964 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:33.780976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.781628 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.781632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:33.781742 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.782283 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:33.782299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:33.785256 (+ 2957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":307,"threads_started":1}
W20260929 11:21:33.785670 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:33.777423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51295 (local address 127.0.97.62:44829)
0929 11:21:33.777604 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:33.777609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:33.777998 (+ 389us) server_negotiation.cc:407] Connection header received
0929 11:21:33.778129 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:33.778132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:33.778186 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:33.778263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:33.779054 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.779747 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.781078 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:33.781469 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:33.781555 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:33.785450 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51295: BlockingRecv error: recv got EOF from 127.0.97.1:51295 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 11:21:34 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:34 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:34.255640 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:34.247532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:39775)
0929 11:21:34.247987 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:34.248007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:34.248113 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:34.248382 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:34.248386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:34.248842 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:34.249039 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:34.249047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.249969 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.249973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:34.250894 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:34.250903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.251626 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.251632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:34.251780 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:34.252366 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:34.252383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:34.255371 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":257,"threads_started":1}
W20260929 11:21:34.255792 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:34.247550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39775 (local address 127.0.97.62:44829)
0929 11:21:34.247804 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:34.247812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:34.248089 (+ 277us) server_negotiation.cc:407] Connection header received
0929 11:21:34.248205 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:34.248209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:34.248267 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:34.248344 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:34.249163 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.249846 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.251020 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.251423 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.251533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:34.255555 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39775: BlockingRecv error: recv got EOF from 127.0.97.2:39775 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 11:21:34 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:34 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:34.705583 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:34.697489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:45243)
0929 11:21:34.697980 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:34.697997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:34.698105 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:34.698421 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:34.698424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:34.698880 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:34.699056 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:34.699064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.699995 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.699999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:34.701188 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:34.701199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.701856 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.701860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:34.701970 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:34.702580 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:34.702597 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:34.705370 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
W20260929 11:21:34.705726 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:34.697500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45243 (local address 127.0.97.62:44829)
0929 11:21:34.697722 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:34.697731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:34.698073 (+ 342us) server_negotiation.cc:407] Connection header received
0929 11:21:34.698215 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:34.698218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:34.698276 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:34.698350 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:34.699170 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.699865 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.701304 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.701704 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.701807 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:34.705524 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45243: BlockingRecv error: recv got EOF from 127.0.97.3:45243 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 11:21:34 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:34 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:34.794595 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:34.786580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:43367)
0929 11:21:34.787071 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:34.787087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:34.787203 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:34.787523 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:34.787526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:34.787888 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:34.788023 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:34.788030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.789103 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.789108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:34.790074 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:34.790086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.790754 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.790759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:34.790871 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:34.791471 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:34.791488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:34.794357 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":293,"threads_started":1}
W20260929 11:21:34.794809 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:34.786605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43367 (local address 127.0.97.62:44829)
0929 11:21:34.786822 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:34.786833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:34.787183 (+ 350us) server_negotiation.cc:407] Connection header received
0929 11:21:34.787320 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:34.787324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:34.787403 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:34.787493 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:34.788165 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.788970 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.790223 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:34.790610 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:34.790707 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:34.794547 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43367: BlockingRecv error: recv got EOF from 127.0.97.1:43367 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 11:21:35 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:35 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:35.265184 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:35.256681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:46089)
0929 11:21:35.257174 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:35.257194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:35.257363 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:35.257704 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:35.257708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:35.258116 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:35.258272 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:35.258280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.259308 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.259314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:35.260369 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:35.260380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.261189 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.261196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:35.261386 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:35.262053 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:35.262079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:35.264907 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
W20260929 11:21:35.265326 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:35.256738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46089 (local address 127.0.97.62:44829)
0929 11:21:35.256954 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:35.256962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:35.257278 (+ 316us) server_negotiation.cc:407] Connection header received
0929 11:21:35.257489 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:35.257494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:35.257569 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:35.257672 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:35.258424 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.259163 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.260516 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.261026 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.261159 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:35.265110 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46089: BlockingRecv error: recv got EOF from 127.0.97.2:46089 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 11:21:35 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:35 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:35.714423 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:35.706457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:33407)
0929 11:21:35.706963 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:35.706976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:35.707112 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:35.707399 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:35.707403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:35.707807 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:35.707936 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:35.707945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.708980 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.708987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:35.709938 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:35.709949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.710805 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.710812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:35.710973 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:35.711569 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:35.711584 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:35.714209 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
W20260929 11:21:35.714634 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:35.706634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33407 (local address 127.0.97.62:44829)
0929 11:21:35.706824 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:35.706829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:35.707047 (+ 218us) server_negotiation.cc:407] Connection header received
0929 11:21:35.707213 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:35.707218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:35.707282 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:35.707374 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:35.708072 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.708843 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.710096 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.710644 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.710769 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:35.714368 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33407: BlockingRecv error: recv got EOF from 127.0.97.3:33407 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 11:21:35 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:35 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:35.803391 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:35.795468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:49587)
0929 11:21:35.795960 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:35.795975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:35.796117 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:35.796435 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:35.796439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:35.796838 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:35.796992 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:35.797000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.798081 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.798086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:35.799055 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:35.799066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.799688 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.799693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:35.799861 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:35.800443 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:35.800459 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:35.803126 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
W20260929 11:21:35.803525 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:35.795541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49587 (local address 127.0.97.62:44829)
0929 11:21:35.795784 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:35.795791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:35.796053 (+ 262us) server_negotiation.cc:407] Connection header received
0929 11:21:35.796241 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:35.796252 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:35.796306 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:35.796405 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:35.797124 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.797938 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.799180 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:35.799557 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:35.799679 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:35.803279 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49587: BlockingRecv error: recv got EOF from 127.0.97.1:49587 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 11:21:36 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:36 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:36.273646 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:36.266005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:48821)
0929 11:21:36.266566 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:36.266587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:36.266695 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:36.266997 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:36.267001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:36.267313 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:36.267468 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:36.267475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.268438 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.268442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:36.269394 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:36.269404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.270095 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.270099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:36.270221 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:36.270873 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:36.270890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:36.273418 (+ 2528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
W20260929 11:21:36.273806 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:36.266056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48821 (local address 127.0.97.62:44829)
0929 11:21:36.266314 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:36.266318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:36.266661 (+ 343us) server_negotiation.cc:407] Connection header received
0929 11:21:36.266788 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:36.266792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:36.266846 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:36.266952 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:36.267580 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.268314 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.269507 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.269866 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.269975 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:36.273607 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48821: BlockingRecv error: recv got EOF from 127.0.97.2:48821 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 29 11:21:36 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:36 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:36.723345 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:36.715299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:47027)
0929 11:21:36.715857 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:36.715873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:36.716041 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:36.716371 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:36.716376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:36.716756 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:36.716907 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:36.716914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.717833 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.717840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:36.718657 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:36.718667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.719329 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.719333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:36.719497 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:36.720094 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:36.720109 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:36.723116 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":292,"threads_started":1}
W20260929 11:21:36.723466 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:36.715497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47027 (local address 127.0.97.62:44829)
0929 11:21:36.715724 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:36.715730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:36.715977 (+ 247us) server_negotiation.cc:407] Connection header received
0929 11:21:36.716175 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:36.716179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:36.716247 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:36.716339 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:36.717043 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.717713 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.718817 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.719204 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.719321 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:36.723297 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47027: BlockingRecv error: recv got EOF from 127.0.97.3:47027 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 11:21:36 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:36 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:36.811944 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:36.804305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:53783)
0929 11:21:36.804838 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:36.804860 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:36.804998 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:36.805304 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:36.805307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:36.805652 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:36.805789 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:36.805796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.806662 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.806665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:36.807592 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:36.807601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.808205 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.808208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:36.808328 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:36.808936 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:36.808957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:36.811615 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
W20260929 11:21:36.812032 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:36.804341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53783 (local address 127.0.97.62:44829)
0929 11:21:36.804536 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:36.804544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:36.804954 (+ 410us) server_negotiation.cc:407] Connection header received
0929 11:21:36.805134 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:36.805138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:36.805195 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:36.805262 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:36.805901 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.806563 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.807704 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:36.808097 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:36.808179 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:36.811797 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53783: BlockingRecv error: recv got EOF from 127.0.97.1:53783 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 11:21:37 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:37 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:37.282567 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:37.274757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:60899)
0929 11:21:37.275291 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:37.275310 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:37.275459 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:37.275803 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:37.275807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:37.276200 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:37.276345 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:37.276353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.277355 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.277362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:37.278272 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:37.278282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.278950 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.278955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:37.279143 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:37.279689 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:37.279705 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:37.282260 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":321,"threads_started":1}
W20260929 11:21:37.282646 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:37.274842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60899 (local address 127.0.97.62:44829)
0929 11:21:37.275038 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:37.275044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:37.275397 (+ 353us) server_negotiation.cc:407] Connection header received
0929 11:21:37.275582 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:37.275587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:37.275670 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:37.275770 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:37.276471 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.277236 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.278404 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.278812 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.278932 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:37.282429 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60899: BlockingRecv error: recv got EOF from 127.0.97.2:60899 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 11:21:37 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:37 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:37.731468 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:37.724147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:58351)
0929 11:21:37.724562 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:37.724577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:37.724699 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:37.724961 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:37.724965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:37.725303 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:37.725448 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:37.725455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.726411 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.726415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:37.727311 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:37.727320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.727930 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.727933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:37.728048 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:37.728691 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:37.728708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:37.731255 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":238,"threads_started":1}
W20260929 11:21:37.731639 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:37.724210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58351 (local address 127.0.97.62:44829)
0929 11:21:37.724430 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:37.724436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:37.724667 (+ 231us) server_negotiation.cc:407] Connection header received
0929 11:21:37.724787 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:37.724790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:37.724845 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:37.724920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:37.725581 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.726306 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.727417 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.727811 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.727906 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:37.731431 (+ 3525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58351: BlockingRecv error: recv got EOF from 127.0.97.3:58351 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 11:21:37 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:37 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:37.821259 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:37.813097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:37401)
0929 11:21:37.813530 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:37.813546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:37.813713 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:37.814060 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:37.814064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:37.814472 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:37.814688 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:37.814700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.816172 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.816178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:37.817236 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:37.817248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.817860 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.817864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:37.817980 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:37.818535 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:37.818557 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:37.821037 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":215,"threads_started":1}
W20260929 11:21:37.821399 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:37.813135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37401 (local address 127.0.97.62:44829)
0929 11:21:37.813342 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:37.813350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:37.813626 (+ 276us) server_negotiation.cc:407] Connection header received
0929 11:21:37.813866 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:37.813870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:37.813938 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:37.814030 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:37.814863 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.816004 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.817353 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:37.817743 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:37.817835 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:37.821209 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37401: BlockingRecv error: recv got EOF from 127.0.97.1:37401 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 11:21:38 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:38 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:38.295295 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:38.283587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:39717)
0929 11:21:38.284172 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:38.284193 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:38.284303 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:38.284648 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:38.284651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:38.285019 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:38.285167 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:38.285175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.286123 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.286126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:38.287017 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:38.287027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.287665 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.287669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:38.287784 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:38.292025 (+ 4241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:38.292041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:38.294957 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":305,"threads_started":1}
W20260929 11:21:38.295403 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:38.283770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39717 (local address 127.0.97.62:44829)
0929 11:21:38.283930 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:38.283934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:38.284270 (+ 336us) server_negotiation.cc:407] Connection header received
0929 11:21:38.284428 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:38.284432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:38.284487 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:38.284565 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:38.285279 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.286009 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.287137 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.287553 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.287650 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:38.295147 (+ 7497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39717: BlockingRecv error: recv got EOF from 127.0.97.2:39717 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 11:21:38 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:38 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:38.740507 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:38.732639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:60945)
0929 11:21:38.733273 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:38.733294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:38.733409 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:38.733772 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:38.733776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:38.734140 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:38.734285 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:38.734293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.735233 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.735236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:38.736153 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:38.736162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.737065 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.737070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:38.737182 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:38.737781 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:38.737797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:38.740266 (+ 2469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":367,"threads_started":1}
W20260929 11:21:38.740690 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:38.732754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60945 (local address 127.0.97.62:44829)
0929 11:21:38.732913 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:38.732920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:38.733376 (+ 456us) server_negotiation.cc:407] Connection header received
0929 11:21:38.733516 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:38.733521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:38.733613 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:38.733724 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:38.734402 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.735105 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.736285 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.736932 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.737047 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:38.740463 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60945: BlockingRecv error: recv got EOF from 127.0.97.3:60945 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 11:21:38 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:38 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:38.830093 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:38.822223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:60991)
0929 11:21:38.822723 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:38.822756 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:38.822865 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:38.823122 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:38.823125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:38.823465 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:38.823622 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:38.823629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.824572 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.824575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:38.825445 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:38.825454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.826087 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.826091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:38.826206 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:38.826857 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:38.826874 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:38.829893 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":304,"threads_started":1}
W20260929 11:21:38.830302 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:38.822265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60991 (local address 127.0.97.62:44829)
0929 11:21:38.822449 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:38.822453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:38.822831 (+ 378us) server_negotiation.cc:407] Connection header received
0929 11:21:38.822952 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:38.822955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:38.823009 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:38.823083 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:38.823736 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.824458 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.825561 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:38.825965 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:38.826059 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:38.830066 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60991: BlockingRecv error: recv got EOF from 127.0.97.1:60991 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 11:21:39 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:39 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:39.304841 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:39.296322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:34171)
0929 11:21:39.296892 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:39.296913 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:39.297024 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:39.297319 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:39.297332 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:39.297711 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:39.297861 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:39.297869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.298988 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.298991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:39.299871 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:39.299880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.300677 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.300683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:39.300872 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:39.301587 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:39.301604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:39.304465 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":332,"threads_started":1}
W20260929 11:21:39.304958 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:39.296390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34171 (local address 127.0.97.62:44829)
0929 11:21:39.296575 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:39.296579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:39.297033 (+ 454us) server_negotiation.cc:407] Connection header received
0929 11:21:39.297116 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:39.297120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:39.297194 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:39.297287 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:39.298014 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.298861 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.299987 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.300410 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.300588 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:39.304720 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34171: BlockingRecv error: recv got EOF from 127.0.97.2:34171 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 11:21:39 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:39 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:39.749475 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:39.741478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:35677)
0929 11:21:39.741961 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:39.741978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:39.742085 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:39.742369 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:39.742372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:39.742735 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:39.742947 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:39.742960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.743953 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.743956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:39.745096 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:39.745106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.745918 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.745924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:39.746118 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:39.746672 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:39.746688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:39.749269 (+ 2581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
W20260929 11:21:39.749707 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:39.741519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35677 (local address 127.0.97.62:44829)
0929 11:21:39.741733 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:39.741742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:39.742057 (+ 315us) server_negotiation.cc:407] Connection header received
0929 11:21:39.742177 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:39.742180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:39.742248 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:39.742323 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:39.743114 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.743829 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.745259 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.745793 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.745892 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:39.749457 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35677: BlockingRecv error: recv got EOF from 127.0.97.3:35677 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 11:21:39 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:39 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:39.838999 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:39.831030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:46303)
0929 11:21:39.831604 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:39.831620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:39.831729 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:39.832063 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:39.832066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:39.832432 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:39.832621 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:39.832628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.833650 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.833654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:39.834726 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:39.834737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.835387 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.835392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:39.835496 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:39.836090 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:39.836107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:39.838764 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
W20260929 11:21:39.839213 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:39.831097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46303 (local address 127.0.97.62:44829)
0929 11:21:39.831303 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:39.831308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:39.831695 (+ 387us) server_negotiation.cc:407] Connection header received
0929 11:21:39.831872 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:39.831877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:39.831937 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:39.832022 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:39.832818 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.833526 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.834846 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:39.835267 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:39.835364 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:39.838964 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46303: BlockingRecv error: recv got EOF from 127.0.97.1:46303 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 11:21:40 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:40 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:40.314280 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:40.305880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:37167)
0929 11:21:40.306916 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:40.306937 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:40.307085 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:40.307432 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:40.307436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:40.307808 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:40.307986 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:40.307993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.309063 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.309067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:40.309915 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:40.309924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.310718 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.310722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:40.310824 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:40.311374 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:40.311390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:40.314051 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":702,"threads_started":1}
W20260929 11:21:40.314462 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:40.305974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37167 (local address 127.0.97.62:44829)
0929 11:21:40.306184 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:40.306192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:40.307036 (+ 844us) server_negotiation.cc:407] Connection header received
0929 11:21:40.307226 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:40.307232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:40.307310 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:40.307387 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:40.308120 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.308950 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.310062 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.310583 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.310703 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:40.314213 (+ 3510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37167: BlockingRecv error: recv got EOF from 127.0.97.2:37167 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 11:21:40 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:40 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:40.758141 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:40.750283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:56261)
0929 11:21:40.750809 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:40.750825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:40.750949 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:40.751214 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:40.751217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:40.751622 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:40.751779 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:40.751787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.752757 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.752760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:40.753693 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:40.753703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.754344 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.754348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:40.754461 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:40.754988 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:40.755004 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:40.757826 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":221,"threads_started":1}
W20260929 11:21:40.758272 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:40.750323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56261 (local address 127.0.97.62:44829)
0929 11:21:40.750535 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:40.750543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:40.750914 (+ 371us) server_negotiation.cc:407] Connection header received
0929 11:21:40.751038 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:40.751041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:40.751099 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:40.751174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:40.751888 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.752664 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.753797 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.754221 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.754326 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:40.758059 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56261: BlockingRecv error: recv got EOF from 127.0.97.3:56261 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 11:21:40 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:40 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:40.847834 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:40.840030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:32945)
0929 11:21:40.840571 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:40.840615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:40.840742 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:40.841110 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:40.841115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:40.841674 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:40.841884 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:40.841896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.842856 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.842859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:40.843710 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:40.843718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.844369 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.844372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:40.844478 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:40.845092 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:40.845108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:40.847631 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":305,"threads_started":1}
W20260929 11:21:40.847985 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:40.840208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32945 (local address 127.0.97.62:44829)
0929 11:21:40.840423 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:40.840431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:40.840693 (+ 262us) server_negotiation.cc:407] Connection header received
0929 11:21:40.840858 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:40.840864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:40.840947 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:40.841055 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:40.842029 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.842740 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.843821 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:40.844232 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:40.844318 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:40.847787 (+ 3469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:32945: BlockingRecv error: recv got EOF from 127.0.97.1:32945 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 11:21:41 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:41 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:41.323350 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:41.315266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:33901)
0929 11:21:41.315787 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:41.315808 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:41.315955 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:41.316344 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:41.316349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:41.316802 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:41.316962 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:41.316970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.318103 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.318107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:41.318993 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:41.319003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.319790 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.319795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:41.319906 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:41.320461 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:41.320477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:41.323143 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
W20260929 11:21:41.323572 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:41.315338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33901 (local address 127.0.97.62:44829)
0929 11:21:41.315551 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:41.315776 (+ 225us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:41.315886 (+ 110us) server_negotiation.cc:407] Connection header received
0929 11:21:41.316097 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:41.316103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:41.316186 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:41.316289 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:41.317064 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.317973 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.319110 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.319558 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.319645 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:41.323320 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33901: BlockingRecv error: recv got EOF from 127.0.97.2:33901 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 11:21:41 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:41 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:41.767097 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:41.759186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:38155)
0929 11:21:41.759618 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:41.759635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:41.759775 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:41.760076 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:41.760082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:41.760466 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:41.760647 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:41.760658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.761666 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.761673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:41.762573 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:41.762583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.763291 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.763297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:41.763514 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:41.764025 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:41.764041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:41.766779 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
W20260929 11:21:41.767196 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:41.759253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38155 (local address 127.0.97.62:44829)
0929 11:21:41.759480 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:41.759485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:41.759709 (+ 224us) server_negotiation.cc:407] Connection header received
0929 11:21:41.759889 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:41.759894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:41.759958 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:41.760047 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:41.760772 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.761522 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.762718 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.763122 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.763255 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:41.766966 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38155: BlockingRecv error: recv got EOF from 127.0.97.3:38155 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 11:21:41 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:41 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:41.857539 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:41.848890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:34847)
0929 11:21:41.849421 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:41.849436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:41.849543 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:41.849829 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:41.849832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:41.850196 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:41.850349 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:41.850356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.851324 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.851328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:41.852339 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:41.852351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.853170 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.853177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:41.853312 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:41.854034 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:41.854073 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:41.857199 (+ 3126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":298,"threads_started":1}
W20260929 11:21:41.857638 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:41.848950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34847 (local address 127.0.97.62:44829)
0929 11:21:41.849116 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:41.849120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:41.849505 (+ 385us) server_negotiation.cc:407] Connection header received
0929 11:21:41.849651 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:41.849655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:41.849709 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:41.849785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:41.850484 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.851211 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.852467 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:41.853002 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:41.853117 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:41.857387 (+ 4270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34847: BlockingRecv error: recv got EOF from 127.0.97.1:34847 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 11:21:42 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:42 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:42.332834 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:42.324206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:48471)
0929 11:21:42.324795 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:42.324814 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:42.324969 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:42.325285 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:42.325290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:42.325714 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:42.325949 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:42.325958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.327406 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.327414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:42.328334 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:42.328344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.329090 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.329096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:42.329259 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:42.329845 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:42.329861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:42.332455 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
W20260929 11:21:42.332887 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:42.324271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48471 (local address 127.0.97.62:44829)
0929 11:21:42.324497 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:42.324505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:42.324911 (+ 406us) server_negotiation.cc:407] Connection header received
0929 11:21:42.325089 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:42.325098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:42.325153 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:42.325261 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:42.326102 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.327279 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.328467 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.328947 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.329082 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:42.332667 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48471: BlockingRecv error: recv got EOF from 127.0.97.2:48471 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 11:21:42 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:42 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:42.775529 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:42.768192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:58245)
0929 11:21:42.768712 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:42.768728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:42.768851 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:42.769131 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:42.769134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:42.769487 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:42.769637 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:42.769644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.770537 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.770541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:42.771425 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:42.771435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.772085 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.772089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:42.772209 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:42.772745 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:42.772762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:42.775245 (+ 2483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":272,"threads_started":1}
W20260929 11:21:42.775729 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:42.768231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58245 (local address 127.0.97.62:44829)
0929 11:21:42.768461 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:42.768470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:42.768803 (+ 333us) server_negotiation.cc:407] Connection header received
0929 11:21:42.768946 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:42.768950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:42.769018 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:42.769095 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:42.769762 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.770437 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.771534 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.771968 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.772054 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:42.775491 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58245: BlockingRecv error: recv got EOF from 127.0.97.3:58245 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 11:21:42 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:42 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:42.866832 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:42.858511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:53019)
0929 11:21:42.859040 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:42.859056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:42.859227 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:42.859639 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:42.859644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:42.860109 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:42.860322 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:42.860342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.861464 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.861467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:42.862407 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:42.862418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.863136 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.863140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:42.863260 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:42.863826 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:42.863843 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:42.866572 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
W20260929 11:21:42.867003 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:42.858610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53019 (local address 127.0.97.62:44829)
0929 11:21:42.858799 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:42.858804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:42.859331 (+ 527us) server_negotiation.cc:407] Connection header received
0929 11:21:42.859402 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:42.859407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:42.859482 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:42.859588 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:42.860466 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.861340 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.862532 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:42.863004 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:42.863107 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:42.866749 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53019: BlockingRecv error: recv got EOF from 127.0.97.1:53019 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 11:21:43 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:43 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:43.346686 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:43.333893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:40149)
0929 11:21:43.334406 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:43.334428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:43.334532 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:43.334847 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:43.334850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:43.335183 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:43.335328 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:43.335336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.336351 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.336354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:43.337354 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:43.337366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.338252 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.338258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:43.338446 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:43.343498 (+ 5052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:43.343519 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:43.346473 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":317,"threads_started":1}
W20260929 11:21:43.347070 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:43.334106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40149 (local address 127.0.97.62:44829)
0929 11:21:43.334303 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:43.334308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:43.334503 (+ 195us) server_negotiation.cc:407] Connection header received
0929 11:21:43.334629 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:43.334633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:43.334717 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:43.334807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:43.335504 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.336247 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.337513 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.338081 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.338243 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:43.346730 (+ 8487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40149: BlockingRecv error: recv got EOF from 127.0.97.2:40149 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 11:21:43 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:43 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:43.784700 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:43.776524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:46787)
0929 11:21:43.777016 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:43.777033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:43.777137 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:43.777478 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:43.777483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:43.777975 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:43.778114 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:43.778122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.779215 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.779221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:43.780144 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:43.780155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.780915 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.780920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:43.781098 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:43.781681 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:43.781698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:43.784366 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":309,"threads_started":1}
W20260929 11:21:43.784830 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:43.776727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46787 (local address 127.0.97.62:44829)
0929 11:21:43.776940 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:43.776946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:43.777115 (+ 169us) server_negotiation.cc:407] Connection header received
0929 11:21:43.777228 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:43.777233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:43.777332 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:43.777445 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:43.778275 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.779072 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.780284 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.780758 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.780902 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:43.784563 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46787: BlockingRecv error: recv got EOF from 127.0.97.3:46787 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 11:21:43 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:43 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:43.875521 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:43.867672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:48179)
0929 11:21:43.868213 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:43.868229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:43.868337 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:43.868637 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:43.868640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:43.869032 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:43.869215 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:43.869223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.870133 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.870137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:43.871035 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:43.871046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.871684 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.871688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:43.871793 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:43.872448 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:43.872465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:43.875278 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":315,"threads_started":1}
W20260929 11:21:43.875691 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:43.867754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48179 (local address 127.0.97.62:44829)
0929 11:21:43.867943 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:43.867948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:43.868298 (+ 350us) server_negotiation.cc:407] Connection header received
0929 11:21:43.868426 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:43.868430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:43.868499 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:43.868577 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:43.869326 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.870025 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.871145 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:43.871566 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:43.871650 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:43.875457 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48179: BlockingRecv error: recv got EOF from 127.0.97.1:48179 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 11:21:44 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:44 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:44.355988 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:44.347649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:51135)
0929 11:21:44.348156 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:44.348178 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:44.348294 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:44.348571 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:44.348575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:44.349019 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:44.349203 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:44.349211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.350155 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.350159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:44.351082 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:44.351093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.351846 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.351851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:44.351983 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:44.352726 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:44.352754 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:44.355731 (+ 2977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":279,"threads_started":1}
W20260929 11:21:44.356184 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:44.347726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51135 (local address 127.0.97.62:44829)
0929 11:21:44.347980 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:44.347986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:44.348256 (+ 270us) server_negotiation.cc:407] Connection header received
0929 11:21:44.348393 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:44.348396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:44.348454 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:44.348530 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:44.349314 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.350034 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.351231 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.351672 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.351778 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:44.355955 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51135: BlockingRecv error: recv got EOF from 127.0.97.2:51135 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 11:21:44 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:44 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:44.792881 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:44.785475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:38773)
0929 11:21:44.785927 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:44.785943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:44.786057 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:44.786301 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:44.786304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:44.786700 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:44.786864 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:44.786871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.787784 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.787787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:44.788713 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:44.788725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.789305 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.789308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:44.789408 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:44.789933 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:44.789948 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:44.792652 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
W20260929 11:21:44.792977 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:44.785636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38773 (local address 127.0.97.62:44829)
0929 11:21:44.785849 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:44.785855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:44.785993 (+ 138us) server_negotiation.cc:407] Connection header received
0929 11:21:44.786132 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:44.786135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:44.786190 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:44.786262 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:44.786979 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.787686 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.788827 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.789189 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.789275 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:44.792780 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38773: BlockingRecv error: recv got EOF from 127.0.97.3:38773 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 11:21:44 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:44 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:44.884958 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:44.876467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:34095)
0929 11:21:44.877021 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:44.877037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:44.877144 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:44.877471 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:44.877474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:44.877914 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:44.878114 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:44.878123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.879444 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.879447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:44.880344 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:44.880354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.881130 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.881134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:44.881260 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:44.881845 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:44.881862 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:44.884677 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":303,"threads_started":1}
W20260929 11:21:44.885149 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:44.876529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34095 (local address 127.0.97.62:44829)
0929 11:21:44.876746 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:44.876750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:44.877121 (+ 371us) server_negotiation.cc:407] Connection header received
0929 11:21:44.877248 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:44.877255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:44.877315 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:44.877434 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:44.878286 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.879321 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.880470 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:44.881013 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:44.881105 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:44.884880 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34095: BlockingRecv error: recv got EOF from 127.0.97.1:34095 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 11:21:45 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:45 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:45.365466 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:45.356996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:55729)
0929 11:21:45.357596 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:45.357615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:45.357771 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:45.358111 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:45.358116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:45.358504 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:45.358647 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:45.358655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.359611 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.359619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:45.360552 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:45.360563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.361267 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.361272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:45.361454 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:45.362052 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:45.362079 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:45.365133 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":365,"threads_started":1}
W20260929 11:21:45.365547 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:45.357060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55729 (local address 127.0.97.62:44829)
0929 11:21:45.357425 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:45.357433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:45.357706 (+ 273us) server_negotiation.cc:407] Connection header received
0929 11:21:45.357893 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:45.357902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:45.357958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:45.358079 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:45.358792 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.359475 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.360704 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.361116 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.361252 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:45.365348 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55729: BlockingRecv error: recv got EOF from 127.0.97.2:55729 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
Sep 29 11:21:45 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:45 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:45.802381 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:45.793785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:41469)
0929 11:21:45.794362 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:45.794377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:45.794482 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:45.794776 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:45.794779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:45.795159 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:45.795307 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:45.795315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.796269 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.796273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:45.797296 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:45.797311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.798342 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.798346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:45.798525 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:45.799209 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:45.799226 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:45.802134 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":349,"threads_started":1}
W20260929 11:21:45.802589 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:45.793804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41469 (local address 127.0.97.62:44829)
0929 11:21:45.793987 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:45.793991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:45.794451 (+ 460us) server_negotiation.cc:407] Connection header received
0929 11:21:45.794567 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:45.794570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:45.794652 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:45.794729 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:45.795415 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.796146 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.797481 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.798080 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.798154 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:45.802348 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41469: BlockingRecv error: recv got EOF from 127.0.97.3:41469 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 11:21:45 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:45 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:45.893656 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:45.885854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:42617)
0929 11:21:45.886356 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:45.886371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:45.886470 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:45.886737 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:45.886740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:45.887115 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:45.887262 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:45.887269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.888169 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.888182 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:45.889081 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:45.889092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.889763 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.889768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:45.889896 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:45.890540 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:45.890557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:45.893312 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":276,"threads_started":1}
W20260929 11:21:45.893718 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:45.885904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42617 (local address 127.0.97.62:44829)
0929 11:21:45.886088 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:45.886092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:45.886438 (+ 346us) server_negotiation.cc:407] Connection header received
0929 11:21:45.886554 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:45.886557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:45.886608 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:45.886700 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:45.887371 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.888043 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.889201 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:45.889594 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:45.889678 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:45.893488 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42617: BlockingRecv error: recv got EOF from 127.0.97.1:42617 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 11:21:46 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:46 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:46.374788 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:46.366554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:48735)
0929 11:21:46.367090 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:46.367107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:46.367246 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:46.367572 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:46.367576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:46.367949 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:46.368083 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:46.368091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.369050 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.369055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:46.369884 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:46.369894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.370553 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.370560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:46.370696 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:46.371296 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:46.371311 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:46.374469 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"spinlock_wait_cycles":30464,"thread_start_us":330,"threads_started":1}
W20260929 11:21:46.374879 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:46.366715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48735 (local address 127.0.97.62:44829)
0929 11:21:46.366887 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:46.366895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:46.367178 (+ 283us) server_negotiation.cc:407] Connection header received
0929 11:21:46.367384 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:46.367389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:46.367443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:46.367534 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:46.368203 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.368933 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.370009 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.370385 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.370500 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:46.374665 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48735: BlockingRecv error: recv got EOF from 127.0.97.2:48735 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 11:21:46 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:46 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:46.810988 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:46.803251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:42783)
0929 11:21:46.803793 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:46.803809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:46.803939 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:46.804218 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:46.804222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:46.804642 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:46.804820 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:46.804828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.805746 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.805749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:46.806633 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:46.806643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.807289 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.807293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:46.807400 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:46.808041 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:46.808059 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:46.810760 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
W20260929 11:21:46.811126 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:46.803305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42783 (local address 127.0.97.62:44829)
0929 11:21:46.803457 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:46.803462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:46.803880 (+ 418us) server_negotiation.cc:407] Connection header received
0929 11:21:46.804031 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:46.804034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:46.804091 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:46.804166 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:46.804938 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.805639 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.806759 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.807135 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.807217 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:46.810940 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42783: BlockingRecv error: recv got EOF from 127.0.97.3:42783 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 11:21:46 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:46 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:46.903214 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:46.894951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:59491)
0929 11:21:46.895436 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:46.895451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:46.895601 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:46.895952 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:46.895957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:46.896341 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:46.896501 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:46.896509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.897750 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.897755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:46.898707 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:46.898718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.899376 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.899382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:46.899545 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:46.900133 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:46.900149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:46.902995 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
W20260929 11:21:46.903426 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:46.895023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59491 (local address 127.0.97.62:44829)
0929 11:21:46.895368 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:46.895374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:46.895532 (+ 158us) server_negotiation.cc:407] Connection header received
0929 11:21:46.895717 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:46.895722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:46.895794 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:46.895922 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:46.896656 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.897627 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.898836 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:46.899234 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:46.899366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:46.903204 (+ 3838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59491: BlockingRecv error: recv got EOF from 127.0.97.1:59491 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Sep 29 11:21:47 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:47 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:47.383898 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:47.375721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:59273)
0929 11:21:47.376213 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:47.376234 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:47.376343 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:47.376642 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:47.376646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:47.377002 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:47.377161 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:47.377169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.378168 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.378173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:47.379073 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:47.379084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.379807 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.379811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:47.379925 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:47.380509 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:47.380526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:47.383541 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":280,"threads_started":1}
W20260929 11:21:47.384079 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:47.375802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59273 (local address 127.0.97.62:44829)
0929 11:21:47.376005 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:47.376009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:47.376307 (+ 298us) server_negotiation.cc:407] Connection header received
0929 11:21:47.376440 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:47.376443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:47.376502 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:47.376630 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:47.377286 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.378011 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.379205 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.379672 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.379767 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:47.383799 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59273: BlockingRecv error: recv got EOF from 127.0.97.2:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 11:21:47 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:47 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:47.819921 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:47.812021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.3:53675)
0929 11:21:47.812614 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:47.812637 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:47.812775 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:47.813146 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:47.813170 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:47.813648 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:47.813882 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:47.813890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.814810 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.814813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:47.815727 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:47.815736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.816409 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.816413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:47.816525 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:47.817102 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:47.817118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:47.819693 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":298,"threads_started":1}
W20260929 11:21:47.820140 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:47.812140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53675 (local address 127.0.97.62:44829)
0929 11:21:47.812340 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:47.812349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:47.812709 (+ 360us) server_negotiation.cc:407] Connection header received
0929 11:21:47.812904 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:47.812908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:47.812980 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:47.813097 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:47.813997 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.814701 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.815859 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.816293 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.816385 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:47.819929 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53675: BlockingRecv error: recv got EOF from 127.0.97.3:53675 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 11:21:47 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:47 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:47.912447 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:47.904432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.1:37433)
0929 11:21:47.905090 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:47.905105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:47.905214 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:47.905490 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:47.905493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:47.905868 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:47.906092 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:47.906103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.907504 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.907508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:47.908371 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:47.908381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.909032 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.909036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:47.909151 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:47.909662 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:47.909678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:47.912205 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":460,"threads_started":1}
W20260929 11:21:47.912631 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:47.904596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37433 (local address 127.0.97.62:44829)
0929 11:21:47.904779 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:47.904784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:47.905190 (+ 406us) server_negotiation.cc:407] Connection header received
0929 11:21:47.905310 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:47.905313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:47.905373 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:47.905450 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:47.906280 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.907361 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.908501 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:47.908926 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:47.909011 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:47.912381 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37433: BlockingRecv error: recv got EOF from 127.0.97.1:37433 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 11:21:48 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:21:48 dist-test-slave-srst krb5kdc[8638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:21:48.397614 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:48.385118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44829 (local address 127.0.97.2:58195)
0929 11:21:48.385618 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:48.385647 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:48.385746 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:48.386038 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:48.386043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:48.386447 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:48.386622 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:48.386630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.387549 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.387553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:48.388764 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:48.388775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.389691 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.389695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:48.389814 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:48.394450 (+ 4636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:48.394469 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:48.397308 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44829: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":300,"threads_started":1}
W20260929 11:21:48.397701 9374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:48.385184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58195 (local address 127.0.97.62:44829)
0929 11:21:48.385371 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:48.385377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:48.385712 (+ 335us) server_negotiation.cc:407] Connection header received
0929 11:21:48.385830 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:48.385834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:48.385887 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:48.385975 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:48.386735 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.387420 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.388915 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.389573 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.389669 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:48.397486 (+ 7817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58195: BlockingRecv error: recv got EOF from 127.0.97.2:58195 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:21:48.671747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9225
I20260929 11:21:48.683549 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:48.672924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.0.97.1:46061)
0929 11:21:48.673496 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:48.673503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:48.673532 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:21:48.673610 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:48.673616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:48.673839 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:48.674030 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:48.675263 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.676329 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.677616 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.678192 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.678300 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:48.680579 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:48.680588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:48.680659 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:48.680692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:48.683054 (+ 2362us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:21:48.683306 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36344: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":289,"threads_started":1}
W20260929 11:21:48.683694 9685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36344: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:21:48.683670 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9225: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:46061: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:21:48.683913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9225
I20260929 11:21:48.700316 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9376
I20260929 11:21:48.711439 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:48.701363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.0.97.2:46679)
0929 11:21:48.701998 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:48.702009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:48.702038 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:21:48.702095 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:48.702101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:48.702330 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:48.702478 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:48.703762 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.704890 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.706022 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.706524 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.706610 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:48.708878 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:48.708886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:48.708898 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:48.708935 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:48.710998 (+ 2063us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:21:48.711206 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37232: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":289,"threads_started":1}
W20260929 11:21:48.711587 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9376: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46679: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:21:48.711751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9376
I20260929 11:21:48.727720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9507
I20260929 11:21:48.739197 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:48.728774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.0.97.3:46301)
0929 11:21:48.729364 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:48.729373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:48.729402 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:21:48.729465 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:48.729472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:48.729701 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:48.729878 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:48.731045 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.732145 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.733388 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:48.733949 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:48.734069 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:48.736298 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:48.736306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:48.736321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:48.736359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:48.738705 (+ 2346us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:21:48.738973 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60122: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":317,"threads_started":1}
W20260929 11:21:48.739312 9687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60122: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:21:48.739310 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9507: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46301: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:21:48.739450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9507
I20260929 11:21:48.755453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9156
I20260929 11:21:48.756815 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:48.870097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9156
2026-09-29T11:21:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21015 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T11:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:48Z Disabled control of system clock
I20260929 11:21:48.945511 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45981
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 11:21:49.272127 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:49.272418 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:49.272500 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:49.279747 9692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:49.279850 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:49.279889 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:49.279956 9692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:49.280002 9692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:49.289047 9692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:49.290350 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:49.292064 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:49.304685 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:49.304774 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:49.305222 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:49.305555 9692 server_base.cc:1034] running on GCE node
I20260929 11:21:49.306293 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:49.307713 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:49.308903 9692 hybrid_clock.cc:648] HybridClock initialized: now 1790680909308850 us; error 43 us; skew 500 ppm
I20260929 11:21:49.309268 9692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:49.311420 9692 webserver.cc:460] Webserver started at http://127.0.97.62:39783/ using document root <none> and password file <none>
I20260929 11:21:49.312096 9692 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:49.312206 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:49.312542 9692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:49.315385 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "bea5dfcf266548199f649d4dd96c5752"
format_stamp: "Formatted at 2026-09-29 11:21:49 on dist-test-slave-srst"
server_key: "b04ec4cfda22cd7dc43206830828b8c0"
server_key_iv: "473b7ae33c2c1f13ccea96d6c63afcb9"
server_key_version: "encryptionkey@0"
I20260929 11:21:49.316179 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "bea5dfcf266548199f649d4dd96c5752"
format_stamp: "Formatted at 2026-09-29 11:21:49 on dist-test-slave-srst"
server_key: "b04ec4cfda22cd7dc43206830828b8c0"
server_key_iv: "473b7ae33c2c1f13ccea96d6c63afcb9"
server_key_version: "encryptionkey@0"
I20260929 11:21:49.321228 9692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:21:49.324658 9706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:49.326102 9692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:49.326282 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "bea5dfcf266548199f649d4dd96c5752"
format_stamp: "Formatted at 2026-09-29 11:21:49 on dist-test-slave-srst"
server_key: "b04ec4cfda22cd7dc43206830828b8c0"
server_key_iv: "473b7ae33c2c1f13ccea96d6c63afcb9"
server_key_version: "encryptionkey@0"
I20260929 11:21:49.326534 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:49.342448 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:49.343286 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:49.343572 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:49.361348 9692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45981
I20260929 11:21:49.361367 9757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45981 every 8 connection(s)
I20260929 11:21:49.362671 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:49.366139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9692
I20260929 11:21:49.366318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:49.366400 9758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:49.367147 388 external_mini_cluster.cc:1428] Setting key 9a64eee5f008e757ee182ca9220292ea
I20260929 11:21:49.379464 9758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752: Bootstrap starting.
I20260929 11:21:49.380978 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:49.369208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.97.62:45981)
0929 11:21:49.370208 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:49.370217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:49.370253 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:21:49.372162 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:49.372184 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:49.372758 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:49.373081 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:49.373650 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:49.374823 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:49.376730 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:49.377433 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:49.377580 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:49.378711 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:49.378718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:49.378731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:49.379226 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:49.379295 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:49.379315 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:49.379400 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:49.379656 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 11:21:49.379970 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":389,"threads_started":1}
I20260929 11:21:49.382512 9758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:49.383514 9758 log.cc:824] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:49.385919 9758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752: No bootstrap required, opened a new log
I20260929 11:21:49.392530 9758 raft_consensus.cc:348] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea5dfcf266548199f649d4dd96c5752" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45981 } }
I20260929 11:21:49.392966 9758 raft_consensus.cc:374] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:49.393066 9758 raft_consensus.cc:729] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea5dfcf266548199f649d4dd96c5752, State: Initialized, Role: FOLLOWER
I20260929 11:21:49.393657 9758 consensus_queue.cc:260] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea5dfcf266548199f649d4dd96c5752" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45981 } }
I20260929 11:21:49.393887 9758 raft_consensus.cc:388] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:49.394017 9758 raft_consensus.cc:482] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:49.394176 9758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:49.396418 9758 raft_consensus.cc:504] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea5dfcf266548199f649d4dd96c5752" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45981 } }
I20260929 11:21:49.396922 9758 leader_election.cc:302] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bea5dfcf266548199f649d4dd96c5752; no voters:
I20260929 11:21:49.397446 9758 leader_election.cc:288] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:49.397591 9763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:49.398314 9763 raft_consensus.cc:686] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [term 1 LEADER]: Becoming Leader. State: Replica: bea5dfcf266548199f649d4dd96c5752, State: Running, Role: LEADER
I20260929 11:21:49.398834 9763 consensus_queue.cc:237] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea5dfcf266548199f649d4dd96c5752" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45981 } }
I20260929 11:21:49.399405 9758 sys_catalog.cc:564] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:49.401916 9765 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bea5dfcf266548199f649d4dd96c5752. Latest consensus state: current_term: 1 leader_uuid: "bea5dfcf266548199f649d4dd96c5752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea5dfcf266548199f649d4dd96c5752" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45981 } } }
I20260929 11:21:49.402344 9765 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:49.401916 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bea5dfcf266548199f649d4dd96c5752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea5dfcf266548199f649d4dd96c5752" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45981 } } }
I20260929 11:21:49.402876 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:49.407899 9769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:49.413070 9769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:49.422042 9769 catalog_manager.cc:1347] Generated new cluster ID: 45c5a79b08d04ec7a5708b1d48ad72f5
I20260929 11:21:49.422174 9769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:49.431042 9769 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:49.432186 9769 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:49.441468 9769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bea5dfcf266548199f649d4dd96c5752: Generated new TSK 0
I20260929 11:21:49.442345 9769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:21:49.452337 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45981
--builtin_ntp_servers=127.0.97.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 11:21:49.782251 9782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:49.782537 9782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:49.782617 9782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:49.789655 9782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:49.789752 9782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:49.789849 9782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:49.798486 9782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:49.799727 9782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:49.801498 9782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:49.814018 9790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:49.814064 9788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:49.814647 9787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:49.815050 9782 server_base.cc:1034] running on GCE node
I20260929 11:21:49.815687 9782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:49.816717 9782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:49.817957 9782 hybrid_clock.cc:648] HybridClock initialized: now 1790680909817910 us; error 35 us; skew 500 ppm
I20260929 11:21:49.818301 9782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:49.820518 9782 webserver.cc:460] Webserver started at http://127.0.97.1:39751/ using document root <none> and password file <none>
I20260929 11:21:49.821254 9782 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:49.821379 9782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:49.821730 9782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:49.824674 9782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "c8a6d6308b17443ea6322c998a963646"
format_stamp: "Formatted at 2026-09-29 11:21:49 on dist-test-slave-srst"
server_key: "ad911463f3750cd6ac1cae4d062b7e34"
server_key_iv: "6b58ce2336edf4554ef0086385eddcc3"
server_key_version: "encryptionkey@0"
I20260929 11:21:49.825491 9782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c8a6d6308b17443ea6322c998a963646"
format_stamp: "Formatted at 2026-09-29 11:21:49 on dist-test-slave-srst"
server_key: "ad911463f3750cd6ac1cae4d062b7e34"
server_key_iv: "6b58ce2336edf4554ef0086385eddcc3"
server_key_version: "encryptionkey@0"
I20260929 11:21:49.830904 9782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 11:21:49.834354 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:49.835539 9782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:21:49.835726 9782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "c8a6d6308b17443ea6322c998a963646"
format_stamp: "Formatted at 2026-09-29 11:21:49 on dist-test-slave-srst"
server_key: "ad911463f3750cd6ac1cae4d062b7e34"
server_key_iv: "6b58ce2336edf4554ef0086385eddcc3"
server_key_version: "encryptionkey@0"
I20260929 11:21:49.835948 9782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:49.858701 9782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:49.859632 9782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:49.859939 9782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:49.860997 9782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:49.862584 9782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:49.862715 9782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:49.862862 9782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:49.862959 9782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:49.896859 9782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46673
I20260929 11:21:49.896932 9908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46673 every 8 connection(s)
I20260929 11:21:49.898223 9782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:49.906996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9782
I20260929 11:21:49.907188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:49.908044 388 external_mini_cluster.cc:1428] Setting key 87bb3e49d95f26fc863684672c01541e
I20260929 11:21:49.910899 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:49.900961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35805 (local address 127.0.97.62:45981)
0929 11:21:49.901390 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:49.901395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:49.902585 (+ 1190us) server_negotiation.cc:407] Connection header received
0929 11:21:49.904311 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:49.904316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:49.904403 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:49.904515 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:49.905517 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:49.906310 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:49.907966 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:49.908456 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:49.908577 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:49.910015 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:49.910020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:49.910023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:49.910038 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:49.910109 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:49.910125 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:49.910221 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:49.910472 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 11:21:49.910556 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260929 11:21:49.911055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45981
--builtin_ntp_servers=127.0.97.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 11:21:49.911912 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:49.901599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45981 (local address 127.0.97.1:35805)
0929 11:21:49.902444 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:49.902478 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:49.904041 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:49.904747 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:49.904754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:49.904838 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:49.905363 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:49.905377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:49.906490 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:49.906495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:49.907822 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:49.907833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:49.908701 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:49.908710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:49.908921 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:49.909717 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:49.909719 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:49.909755 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:49.909827 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:49.910221 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:49.910233 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:49.910243 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:21:49.910466 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 11:21:49.910860 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
I20260929 11:21:49.913535 9909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45981
I20260929 11:21:49.913868 9909 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:49.914789 9909 heartbeater.cc:511] Master 127.0.97.62:45981 requested a full tablet report, sending...
I20260929 11:21:49.917131 9723 ts_manager.cc:194] Registered new tserver with Master: c8a6d6308b17443ea6322c998a963646 (127.0.97.1:46673)
I20260929 11:21:49.918668 9723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35805
W20260929 11:21:50.230680 9913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:50.231015 9913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:50.231174 9913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:50.238462 9913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:50.238600 9913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:50.238790 9913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:50.247745 9913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:50.249032 9913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:50.250705 9913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:50.263299 9918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:50.263343 9921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:50.263954 9919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:50.264072 9913 server_base.cc:1034] running on GCE node
I20260929 11:21:50.264706 9913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:50.265506 9913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:50.266750 9913 hybrid_clock.cc:648] HybridClock initialized: now 1790680910266701 us; error 40 us; skew 500 ppm
I20260929 11:21:50.267091 9913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:50.269275 9913 webserver.cc:460] Webserver started at http://127.0.97.2:46359/ using document root <none> and password file <none>
I20260929 11:21:50.269912 9913 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:50.270026 9913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:50.270361 9913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:50.273108 9913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "d9de6c76fc4e4e9e82d3b148747eabc8"
format_stamp: "Formatted at 2026-09-29 11:21:50 on dist-test-slave-srst"
server_key: "797705d12d4cdb495ec6cebae617e07b"
server_key_iv: "c2a6d0bf6cdabea69c48de311c54ab8a"
server_key_version: "encryptionkey@0"
I20260929 11:21:50.273867 9913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d9de6c76fc4e4e9e82d3b148747eabc8"
format_stamp: "Formatted at 2026-09-29 11:21:50 on dist-test-slave-srst"
server_key: "797705d12d4cdb495ec6cebae617e07b"
server_key_iv: "c2a6d0bf6cdabea69c48de311c54ab8a"
server_key_version: "encryptionkey@0"
I20260929 11:21:50.278632 9913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:21:50.282008 9927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:50.283275 9913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:21:50.283461 9913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "d9de6c76fc4e4e9e82d3b148747eabc8"
format_stamp: "Formatted at 2026-09-29 11:21:50 on dist-test-slave-srst"
server_key: "797705d12d4cdb495ec6cebae617e07b"
server_key_iv: "c2a6d0bf6cdabea69c48de311c54ab8a"
server_key_version: "encryptionkey@0"
I20260929 11:21:50.283681 9913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:50.304447 9913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:50.305344 9913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:50.305651 9913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:50.306775 9913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:50.308290 9913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:50.308404 9913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:50.308565 9913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:50.308706 9913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 11:21:50.344342 9913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44135
I20260929 11:21:50.344413 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44135 every 8 connection(s)
I20260929 11:21:50.345757 9913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:50.354417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 9913
I20260929 11:21:50.354604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:50.355031 388 external_mini_cluster.cc:1428] Setting key 535d2ffb0766f16374ece490cc3dca51
I20260929 11:21:50.357877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45981
--builtin_ntp_servers=127.0.97.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 11:21:50.358718 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:50.348291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48511 (local address 127.0.97.62:45981)
0929 11:21:50.348492 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:50.348497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:50.349998 (+ 1501us) server_negotiation.cc:407] Connection header received
0929 11:21:50.351809 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:50.351817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:50.351909 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:50.352049 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:50.353221 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.353945 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.355690 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.356251 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.356416 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:50.357887 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:50.357897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:50.357904 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:50.357928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:50.357972 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:50.357990 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:50.358099 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:50.358409 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 11:21:50.358485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:21:50.360304 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:50.348914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45981 (local address 127.0.97.2:48511)
0929 11:21:50.349822 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:50.349870 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:50.351540 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:50.352281 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:50.352289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:50.352372 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:50.353047 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:50.353064 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.354140 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.354147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:50.355544 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:50.355557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.356491 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.356496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:50.356842 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:50.357570 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:50.357572 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:50.357612 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:50.357688 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:50.358159 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:50.358172 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:50.358183 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:21:50.358365 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 11:21:50.358816 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":257,"threads_started":1}
I20260929 11:21:50.362010 10040 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45981
I20260929 11:21:50.362329 10040 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:50.363046 10040 heartbeater.cc:511] Master 127.0.97.62:45981 requested a full tablet report, sending...
I20260929 11:21:50.364382 9723 ts_manager.cc:194] Registered new tserver with Master: d9de6c76fc4e4e9e82d3b148747eabc8 (127.0.97.2:44135)
I20260929 11:21:50.365257 9723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48511
W20260929 11:21:50.675874 10043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:50.676200 10043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:50.676389 10043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:50.683434 10043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:50.683562 10043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:50.683748 10043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:50.692449 10043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:50.693754 10043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:50.695446 10043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:50.707988 10048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:50.708009 10051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:50.707962 10049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:50.708882 10043 server_base.cc:1034] running on GCE node
I20260929 11:21:50.709543 10043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:50.710536 10043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:50.711720 10043 hybrid_clock.cc:648] HybridClock initialized: now 1790680910711694 us; error 48 us; skew 500 ppm
I20260929 11:21:50.712015 10043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:50.714179 10043 webserver.cc:460] Webserver started at http://127.0.97.3:44641/ using document root <none> and password file <none>
I20260929 11:21:50.714874 10043 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:50.714975 10043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:50.715296 10043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:50.718133 10043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "acd110b47bb047a1b4545e2d596f7f24"
format_stamp: "Formatted at 2026-09-29 11:21:50 on dist-test-slave-srst"
server_key: "a562d00fa8d5244f0a3e8c2e52a7f5eb"
server_key_iv: "932f2caaece0d8884c2f0a750841cc20"
server_key_version: "encryptionkey@0"
I20260929 11:21:50.718879 10043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "acd110b47bb047a1b4545e2d596f7f24"
format_stamp: "Formatted at 2026-09-29 11:21:50 on dist-test-slave-srst"
server_key: "a562d00fa8d5244f0a3e8c2e52a7f5eb"
server_key_iv: "932f2caaece0d8884c2f0a750841cc20"
server_key_version: "encryptionkey@0"
I20260929 11:21:50.723688 10043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 11:21:50.726972 10057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:50.728406 10043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:21:50.728569 10043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "acd110b47bb047a1b4545e2d596f7f24"
format_stamp: "Formatted at 2026-09-29 11:21:50 on dist-test-slave-srst"
server_key: "a562d00fa8d5244f0a3e8c2e52a7f5eb"
server_key_iv: "932f2caaece0d8884c2f0a750841cc20"
server_key_version: "encryptionkey@0"
I20260929 11:21:50.728868 10043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:50.746755 10043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:50.747572 10043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:50.747855 10043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:50.748791 10043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:50.750202 10043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:50.750332 10043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:50.750438 10043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:50.750514 10043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:50.782585 10043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40767
I20260929 11:21:50.782675 10169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40767 every 8 connection(s)
I20260929 11:21:50.783959 10043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:50.793934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10043
I20260929 11:21:50.794106 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:50.794560 388 external_mini_cluster.cc:1428] Setting key 8f48fa2582ff0e652014a604788ddfc1
I20260929 11:21:50.797941 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:50.786757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52401 (local address 127.0.97.62:45981)
0929 11:21:50.786958 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:50.786963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:50.788416 (+ 1453us) server_negotiation.cc:407] Connection header received
0929 11:21:50.790503 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:50.790510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:50.790614 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:50.790739 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:50.791830 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.792711 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.794599 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.795207 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.795378 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:50.796987 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:50.796996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:50.797001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:50.797032 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:50.797065 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:50.797080 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:50.797205 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:50.797447 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 11:21:50.797552 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:21:50.798985 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:50.787282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45981 (local address 127.0.97.3:52401)
0929 11:21:50.788241 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:50.788278 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:50.790219 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:50.790982 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:50.790991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:50.791078 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:50.791657 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:50.791674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.792872 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.792886 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:50.794402 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:50.794452 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.795461 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.795471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:50.795763 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:50.796581 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:50.796584 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:50.796689 (+ 105us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:50.796770 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:50.797202 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:50.797215 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:50.797227 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 11:21:50.797438 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 11:21:50.797868 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":311,"threads_started":1}
I20260929 11:21:50.800240 10170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45981
I20260929 11:21:50.800501 10170 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:50.801364 10170 heartbeater.cc:511] Master 127.0.97.62:45981 requested a full tablet report, sending...
I20260929 11:21:50.802598 9723 ts_manager.cc:194] Registered new tserver with Master: acd110b47bb047a1b4545e2d596f7f24 (127.0.97.3:40767)
I20260929 11:21:50.803361 9723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52401
I20260929 11:21:50.809551 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:21:50.815572 9911 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:50.813966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.0.97.62:45981)
0929 11:21:50.814197 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:50.814205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:50.814559 (+ 354us) server_negotiation.cc:407] Connection header received
0929 11:21:50.814750 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:50.814754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:50.814825 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:50.814923 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:50.815312 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53540: BlockingRecv error: recv got EOF from 127.0.0.1:53540 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:21:50.816453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9782
I20260929 11:21:50.824179 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:50.817390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.97.1:46673)
0929 11:21:50.817923 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:50.817930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:50.817955 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:21:50.818005 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:50.818012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:50.818207 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:50.818324 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:50.818689 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.819675 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.821159 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.821711 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.821820 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:50.822890 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:50.822896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:50.822927 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:50.823485 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:50.823527 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:50.823544 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:50.823625 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:50.823806 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 11:21:50.823902 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260929 11:21:50.825492 9904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:50.905085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9782
I20260929 11:21:50.921178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9913
I20260929 11:21:50.928965 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:50.922070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.0.97.2:44135)
0929 11:21:50.922612 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:50.922619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:50.922646 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:21:50.922706 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:50.922712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:50.922939 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:50.923064 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:50.923390 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.924372 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.925737 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:50.926301 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:50.926395 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:50.927606 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:50.927613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:50.927626 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:50.928223 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:50.928267 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:50.928285 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:50.928368 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:50.928506 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 11:21:50.928668 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260929 11:21:50.930526 10035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:51.010149 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9913
I20260929 11:21:51.026010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10043
I20260929 11:21:51.035246 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:51.027058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.97.3:40767)
0929 11:21:51.027735 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:51.027742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:51.027774 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:21:51.027833 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:51.027839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:51.028068 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:51.028280 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:51.028805 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:51.030067 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:51.031476 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:51.032122 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:51.032274 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:51.033689 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:51.033698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:51.033713 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:51.034343 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:51.034390 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:51.034410 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:51.034557 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:51.034736 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 11:21:51.034874 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":355,"threads_started":1}
I20260929 11:21:51.036544 10165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:51.115576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10043
I20260929 11:21:51.131373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 9692
I20260929 11:21:51.132490 9753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:51.231827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 9692
2026-09-29T11:21:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2355 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T11:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:51Z Disabled control of system clock
I20260929 11:21:51.300339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37351
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 11:21:51.619990 10189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:51.620287 10189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:51.620373 10189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:51.627492 10189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:51.627595 10189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:51.627632 10189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:51.627679 10189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:51.627715 10189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:51.636560 10189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:51.637828 10189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:51.639526 10189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:51.651022 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:51.651104 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:51.650991 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:51.654224 10189 server_base.cc:1034] running on GCE node
I20260929 11:21:51.655025 10189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:51.656464 10189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:51.657706 10189 hybrid_clock.cc:648] HybridClock initialized: now 1790680911657676 us; error 68 us; skew 500 ppm
I20260929 11:21:51.658061 10189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:51.660321 10189 webserver.cc:460] Webserver started at http://127.0.97.62:33873/ using document root <none> and password file <none>
I20260929 11:21:51.661072 10189 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:51.661202 10189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:51.661526 10189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:51.664350 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "04fa34f4d8ce49409ec88741f0a3fc28"
format_stamp: "Formatted at 2026-09-29 11:21:51 on dist-test-slave-srst"
server_key: "5291ae8ede309e7c201f5a008e94dd4f"
server_key_iv: "4352c931487611d1e7e75b47be882673"
server_key_version: "encryptionkey@0"
I20260929 11:21:51.665272 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "04fa34f4d8ce49409ec88741f0a3fc28"
format_stamp: "Formatted at 2026-09-29 11:21:51 on dist-test-slave-srst"
server_key: "5291ae8ede309e7c201f5a008e94dd4f"
server_key_iv: "4352c931487611d1e7e75b47be882673"
server_key_version: "encryptionkey@0"
I20260929 11:21:51.670480 10189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:21:51.673813 10203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:51.675120 10189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:21:51.675305 10189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "04fa34f4d8ce49409ec88741f0a3fc28"
format_stamp: "Formatted at 2026-09-29 11:21:51 on dist-test-slave-srst"
server_key: "5291ae8ede309e7c201f5a008e94dd4f"
server_key_iv: "4352c931487611d1e7e75b47be882673"
server_key_version: "encryptionkey@0"
I20260929 11:21:51.675534 10189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:51.681128 10189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:51.681879 10189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:51.682169 10189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:51.700484 10189 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37351
I20260929 11:21:51.700517 10254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37351 every 8 connection(s)
I20260929 11:21:51.701841 10189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:51.705592 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:51.709810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10189
I20260929 11:21:51.710016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:51.710765 388 external_mini_cluster.cc:1428] Setting key 78bb84a4f41ab4560a35702aa4bef765
I20260929 11:21:51.717123 10255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28: Bootstrap starting.
I20260929 11:21:51.719237 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:51.712770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.0.97.62:37351)
0929 11:21:51.713737 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:51.713748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:51.713794 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:21:51.715713 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:51.715736 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:51.716142 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:51.716556 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:51.717667 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:51.717673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:51.717685 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:51.717710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:51.717764 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:51.717779 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:51.718000 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:51.718126 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 11:21:51.718355 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":366,"threads_started":1}
I20260929 11:21:51.720388 10255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:51.721439 10255 log.cc:824] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:51.723963 10255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28: No bootstrap required, opened a new log
I20260929 11:21:51.731457 10255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37351 } }
I20260929 11:21:51.731853 10255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:51.731943 10255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fa34f4d8ce49409ec88741f0a3fc28, State: Initialized, Role: FOLLOWER
I20260929 11:21:51.732537 10255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37351 } }
I20260929 11:21:51.732859 10255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:51.733033 10255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:51.733235 10255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:51.735371 10255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37351 } }
I20260929 11:21:51.735862 10255 leader_election.cc:302] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04fa34f4d8ce49409ec88741f0a3fc28; no voters:
I20260929 11:21:51.736410 10255 leader_election.cc:288] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:51.736578 10260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:51.737321 10260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [term 1 LEADER]: Becoming Leader. State: Replica: 04fa34f4d8ce49409ec88741f0a3fc28, State: Running, Role: LEADER
I20260929 11:21:51.737828 10260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37351 } }
I20260929 11:21:51.738579 10255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:51.741389 10262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04fa34f4d8ce49409ec88741f0a3fc28. Latest consensus state: current_term: 1 leader_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37351 } } }
I20260929 11:21:51.741735 10262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:51.741511 10261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa34f4d8ce49409ec88741f0a3fc28" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37351 } } }
I20260929 11:21:51.741927 10261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:51.743723 10268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:51.749714 10268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:51.758626 10268 catalog_manager.cc:1347] Generated new cluster ID: 9bd81f31da0346a1b08d52e609498608
I20260929 11:21:51.758744 10268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:51.769553 10268 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:51.770033 10268 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:51.785131 10268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04fa34f4d8ce49409ec88741f0a3fc28: Generated new TSK 0
I20260929 11:21:51.786192 10268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:21:51.791708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37351
--builtin_ntp_servers=127.0.97.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 11:21:52.120101 10279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:52.120430 10279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:52.120637 10279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:52.127985 10279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:52.128124 10279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:52.128326 10279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:52.137624 10279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:52.139034 10279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:52.140945 10279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:52.154006 10284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:52.154146 10285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:52.154049 10287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:52.154440 10279 server_base.cc:1034] running on GCE node
I20260929 11:21:52.155264 10279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:52.156190 10279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:52.157387 10279 hybrid_clock.cc:648] HybridClock initialized: now 1790680912157342 us; error 41 us; skew 500 ppm
I20260929 11:21:52.157714 10279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:52.159864 10279 webserver.cc:460] Webserver started at http://127.0.97.1:35863/ using document root <none> and password file <none>
I20260929 11:21:52.160562 10279 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:52.160730 10279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:52.161062 10279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:52.163882 10279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "eb3bb10448b74995af2180cee3be6ad5"
format_stamp: "Formatted at 2026-09-29 11:21:52 on dist-test-slave-srst"
server_key: "22e7d1dd6a58eca034e3a77f66a34801"
server_key_iv: "8f20f53159b281e33203547cb4668e3e"
server_key_version: "encryptionkey@0"
I20260929 11:21:52.164723 10279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eb3bb10448b74995af2180cee3be6ad5"
format_stamp: "Formatted at 2026-09-29 11:21:52 on dist-test-slave-srst"
server_key: "22e7d1dd6a58eca034e3a77f66a34801"
server_key_iv: "8f20f53159b281e33203547cb4668e3e"
server_key_version: "encryptionkey@0"
I20260929 11:21:52.169741 10279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 11:21:52.173105 10293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:52.174378 10279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:21:52.174561 10279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "eb3bb10448b74995af2180cee3be6ad5"
format_stamp: "Formatted at 2026-09-29 11:21:52 on dist-test-slave-srst"
server_key: "22e7d1dd6a58eca034e3a77f66a34801"
server_key_iv: "8f20f53159b281e33203547cb4668e3e"
server_key_version: "encryptionkey@0"
I20260929 11:21:52.174784 10279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:52.180424 10279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:52.181209 10279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:52.181483 10279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:52.182361 10279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:52.183682 10279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:52.183795 10279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:52.183923 10279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:52.184008 10279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:52.216861 10279 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42433
I20260929 11:21:52.216907 10405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42433 every 8 connection(s)
I20260929 11:21:52.218364 10279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:52.225368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10279
I20260929 11:21:52.225584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:52.226390 388 external_mini_cluster.cc:1428] Setting key 08cdfbf74072c68a1ec98d554c89622b
I20260929 11:21:52.226701 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:52.221027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47139 (local address 127.0.97.62:37351)
0929 11:21:52.221440 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:52.221444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:52.222727 (+ 1283us) server_negotiation.cc:407] Connection header received
0929 11:21:52.224337 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:52.224341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:52.224389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:52.224493 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:52.225915 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:52.225920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:52.225925 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:52.225945 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:52.225996 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:52.226014 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:52.226140 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:52.226312 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 11:21:52.226378 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":216,"threads_started":1}
I20260929 11:21:52.227619 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:52.221632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37351 (local address 127.0.97.1:47139)
0929 11:21:52.222511 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:52.222546 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:52.224107 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:52.224766 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:52.224774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:52.224864 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:52.225587 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:52.225589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:52.225625 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:52.225748 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:52.226121 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:52.226127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:52.226133 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 11:21:52.226343 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 11:21:52.226592 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":264,"threads_started":1}
I20260929 11:21:52.229164 10406 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37351
I20260929 11:21:52.229293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37351
--builtin_ntp_servers=127.0.97.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 11:21:52.229489 10406 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:52.230012 10406 heartbeater.cc:511] Master 127.0.97.62:37351 requested a full tablet report, sending...
I20260929 11:21:52.231957 10220 ts_manager.cc:194] Registered new tserver with Master: eb3bb10448b74995af2180cee3be6ad5 (127.0.97.1:42433)
W20260929 11:21:52.550376 10410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:52.550638 10410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:52.550724 10410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:52.557902 10410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:52.558017 10410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:52.558125 10410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:52.567046 10410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:52.568254 10410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:52.570068 10410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:52.584081 10418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:52.584054 10415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:52.584118 10416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:52.584954 10410 server_base.cc:1034] running on GCE node
I20260929 11:21:52.585515 10410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:52.586347 10410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:52.587548 10410 hybrid_clock.cc:648] HybridClock initialized: now 1790680912587514 us; error 34 us; skew 500 ppm
I20260929 11:21:52.587872 10410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:52.590169 10410 webserver.cc:460] Webserver started at http://127.0.97.2:42187/ using document root <none> and password file <none>
I20260929 11:21:52.590849 10410 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:52.590968 10410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:52.591284 10410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:52.594432 10410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "655e022974424e2781da5be891bbdbdc"
format_stamp: "Formatted at 2026-09-29 11:21:52 on dist-test-slave-srst"
server_key: "0b897dc8e5e3f915395dc859d481f9ed"
server_key_iv: "fa4f0ecf8910eef92d8298d28abfc8af"
server_key_version: "encryptionkey@0"
I20260929 11:21:52.595240 10410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "655e022974424e2781da5be891bbdbdc"
format_stamp: "Formatted at 2026-09-29 11:21:52 on dist-test-slave-srst"
server_key: "0b897dc8e5e3f915395dc859d481f9ed"
server_key_iv: "fa4f0ecf8910eef92d8298d28abfc8af"
server_key_version: "encryptionkey@0"
I20260929 11:21:52.600392 10410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:21:52.603686 10424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:52.605073 10410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:52.605253 10410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "655e022974424e2781da5be891bbdbdc"
format_stamp: "Formatted at 2026-09-29 11:21:52 on dist-test-slave-srst"
server_key: "0b897dc8e5e3f915395dc859d481f9ed"
server_key_iv: "fa4f0ecf8910eef92d8298d28abfc8af"
server_key_version: "encryptionkey@0"
I20260929 11:21:52.605507 10410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:52.611248 10410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:52.612125 10410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:52.612340 10410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:52.613217 10410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:52.614535 10410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:52.614616 10410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:52.614696 10410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:52.614748 10410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:52.646631 10410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45617
I20260929 11:21:52.646680 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45617 every 8 connection(s)
I20260929 11:21:52.647915 10410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:52.650632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10410
I20260929 11:21:52.650835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:52.651259 388 external_mini_cluster.cc:1428] Setting key 21a357e2cfc9d33f1377e273feabd3c7
I20260929 11:21:52.654268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37351
--builtin_ntp_servers=127.0.97.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 11:21:52.656978 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:52.650479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60883 (local address 127.0.97.62:37351)
0929 11:21:52.650683 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:52.650689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:52.652371 (+ 1682us) server_negotiation.cc:407] Connection header received
0929 11:21:52.654513 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:52.654521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:52.654584 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:52.654654 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:52.656300 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:52.656305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:52.656308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:52.656322 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:52.656351 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:52.656362 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:52.656460 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:52.656718 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 11:21:52.656766 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:21:52.658190 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:52.651126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37351 (local address 127.0.97.2:60883)
0929 11:21:52.652205 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:52.652252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:52.654187 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:52.655150 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:52.655163 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:52.655287 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:52.655965 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:52.655970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:52.656026 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:52.656134 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:52.656467 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:52.656479 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:52.656489 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:21:52.656800 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 11:21:52.657048 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260929 11:21:52.659484 10537 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37351
I20260929 11:21:52.659768 10537 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:52.660403 10537 heartbeater.cc:511] Master 127.0.97.62:37351 requested a full tablet report, sending...
I20260929 11:21:52.661759 10220 ts_manager.cc:194] Registered new tserver with Master: 655e022974424e2781da5be891bbdbdc (127.0.97.2:45617)
W20260929 11:21:52.973451 10540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:52.973779 10540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:52.973954 10540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:52.981117 10540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:52.981235 10540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:52.981415 10540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:52.989782 10540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:52.991000 10540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:52.992658 10540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:53.005456 10548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:53.005452 10545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:53.005548 10546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:53.006011 10540 server_base.cc:1034] running on GCE node
I20260929 11:21:53.006649 10540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:53.007529 10540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:53.008754 10540 hybrid_clock.cc:648] HybridClock initialized: now 1790680913008718 us; error 51 us; skew 500 ppm
I20260929 11:21:53.009130 10540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:21:53.011235 10540 webserver.cc:460] Webserver started at http://127.0.97.3:37251/ using document root <none> and password file <none>
I20260929 11:21:53.011873 10540 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:53.012034 10540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:53.012385 10540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:53.015172 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "a697f0c77aab48f1abb97cacbc509a2d"
format_stamp: "Formatted at 2026-09-29 11:21:53 on dist-test-slave-srst"
server_key: "01870c985e702d6f206d0dfc27248fff"
server_key_iv: "3d3537d7a0ae8997c425f6e3325a6dc7"
server_key_version: "encryptionkey@0"
I20260929 11:21:53.015928 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a697f0c77aab48f1abb97cacbc509a2d"
format_stamp: "Formatted at 2026-09-29 11:21:53 on dist-test-slave-srst"
server_key: "01870c985e702d6f206d0dfc27248fff"
server_key_iv: "3d3537d7a0ae8997c425f6e3325a6dc7"
server_key_version: "encryptionkey@0"
I20260929 11:21:53.020992 10540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:21:53.025861 10554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:53.027798 10540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:21:53.027966 10540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "a697f0c77aab48f1abb97cacbc509a2d"
format_stamp: "Formatted at 2026-09-29 11:21:53 on dist-test-slave-srst"
server_key: "01870c985e702d6f206d0dfc27248fff"
server_key_iv: "3d3537d7a0ae8997c425f6e3325a6dc7"
server_key_version: "encryptionkey@0"
I20260929 11:21:53.028275 10540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:53.036406 10540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:53.037459 10540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:53.037756 10540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:53.038933 10540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:53.040802 10540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:53.040911 10540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:53.041020 10540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:53.041101 10540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:53.076839 10540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34673
I20260929 11:21:53.076932 10666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34673 every 8 connection(s)
I20260929 11:21:53.078747 10540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:53.086879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10540
I20260929 11:21:53.087055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:53.087617 388 external_mini_cluster.cc:1428] Setting key 2bad26b2745a07450a4727d60d0ea5d5
I20260929 11:21:53.089979 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:53.082387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37351 (local address 127.0.97.3:47081)
0929 11:21:53.083489 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:53.083544 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:53.086173 (+ 2629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:53.087080 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:53.087091 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:53.087208 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:53.087955 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:21:53.087957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:53.087992 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:21:53.088111 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:53.088480 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:53.088486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:53.088491 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:21:53.088771 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 11:21:53.089024 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260929 11:21:53.090608 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:53.081857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47081 (local address 127.0.97.62:37351)
0929 11:21:53.082101 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:53.082109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:53.083669 (+ 1560us) server_negotiation.cc:407] Connection header received
0929 11:21:53.086492 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:53.086514 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:53.086580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:53.086693 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:53.088321 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:53.088326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:53.088330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:53.088345 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:53.088384 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:53.088398 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:53.088508 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:53.090362 (+ 1854us) server_negotiation.cc:299] Negotiation successful
0929 11:21:53.090431 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:21:53.091562 10667 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37351
I20260929 11:21:53.091791 10667 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:53.092272 10667 heartbeater.cc:511] Master 127.0.97.62:37351 requested a full tablet report, sending...
I20260929 11:21:53.093631 10220 ts_manager.cc:194] Registered new tserver with Master: a697f0c77aab48f1abb97cacbc509a2d (127.0.97.3:34673)
I20260929 11:21:53.102558 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:21:53.108351 10408 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:21:53.106767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.97.62:37351)
0929 11:21:53.106922 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:53.106926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:53.107426 (+ 500us) server_negotiation.cc:407] Connection header received
0929 11:21:53.107623 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:53.107629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:53.107683 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:53.107793 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:53.108109 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40680: BlockingRecv error: recv got EOF from 127.0.0.1:40680 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:21:53.109090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10279
I20260929 11:21:53.112766 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:53.110178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.0.97.1:42433)
0929 11:21:53.110654 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:53.110661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:53.110685 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:21:53.110752 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:53.110757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:53.110966 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:53.111083 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:53.111949 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:53.111955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:53.111966 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:53.111985 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:53.112027 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:53.112042 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:53.112141 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:53.112340 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 11:21:53.112458 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
I20260929 11:21:53.113904 10401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:53.192504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10279
I20260929 11:21:53.208899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10410
I20260929 11:21:53.212297 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:53.209827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.97.2:45617)
0929 11:21:53.210339 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:53.210346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:53.210371 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:21:53.210429 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:53.210435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:53.210649 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:53.210767 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:53.211651 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:53.211657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:53.211667 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:53.211683 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:53.211719 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:53.211732 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:53.211829 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:53.211904 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 11:21:53.211979 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260929 11:21:53.213567 10532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:53.293546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10410
I20260929 11:21:53.309111 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10540
I20260929 11:21:53.312866 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:53.310117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.97.3:34673)
0929 11:21:53.310759 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:53.310765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:53.310794 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:21:53.310854 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:53.310860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:53.311073 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:53.311214 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:53.312081 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:53.312087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:53.312097 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:21:53.312113 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:53.312167 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:21:53.312180 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:53.312309 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:53.312357 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 11:21:53.312464 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":337,"threads_started":1}
I20260929 11:21:53.314097 10662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:53.392889 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10540
I20260929 11:21:53.408517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10189
I20260929 11:21:53.409745 10250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:53.507748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10189
2026-09-29T11:21:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2275 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:53 dist-test-slave-srst krb5kdc[10685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:53 dist-test-slave-srst krb5kdc[10685](info): set up 2 sockets
Sep 29 11:21:53 dist-test-slave-srst krb5kdc[10685](info): commencing operation
krb5kdc: starting...
W20260929 11:21:54.573248 388 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:54 dist-test-slave-srst krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:21:54.788036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33611
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:55.099570 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:55.099884 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:55.100032 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:55.107123 10698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:21:55.107251 10698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:55.107349 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:55.107439 10698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:21:55.107517 10698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:21:55.116849 10698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:55.118166 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:55.119441 10698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:55.130393 10706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:55.130460 10703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:55.130422 10704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:55.131016 10698 server_base.cc:1034] running on GCE node
I20260929 11:21:55.131820 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:55.133162 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:55.134380 10698 hybrid_clock.cc:648] HybridClock initialized: now 1790680915134349 us; error 42 us; skew 500 ppm
I20260929 11:21:55.134709 10698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:55 dist-test-slave-srst krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:55.137863 10698 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:21:55.139245 10698 webserver.cc:460] Webserver started at http://127.0.97.62:35555/ using document root <none> and password file <none>
I20260929 11:21:55.139906 10698 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:55.140049 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:55.140358 10698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:55.143262 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "22f2366871c047c28665415113ff6193"
format_stamp: "Formatted at 2026-09-29 11:21:55 on dist-test-slave-srst"
server_key: "f8a29bfa3d8ef939069307302c34c1f8"
server_key_iv: "70cca6ec18363dbfd4988015c18796aa"
server_key_version: "encryptionkey@0"
I20260929 11:21:55.144001 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "22f2366871c047c28665415113ff6193"
format_stamp: "Formatted at 2026-09-29 11:21:55 on dist-test-slave-srst"
server_key: "f8a29bfa3d8ef939069307302c34c1f8"
server_key_iv: "70cca6ec18363dbfd4988015c18796aa"
server_key_version: "encryptionkey@0"
I20260929 11:21:55.149040 10698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 11:21:55.152036 10713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:55.153388 10698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 11:21:55.153586 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "22f2366871c047c28665415113ff6193"
format_stamp: "Formatted at 2026-09-29 11:21:55 on dist-test-slave-srst"
server_key: "f8a29bfa3d8ef939069307302c34c1f8"
server_key_iv: "70cca6ec18363dbfd4988015c18796aa"
server_key_version: "encryptionkey@0"
I20260929 11:21:55.153834 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:55.171407 10698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:55.175534 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:55.175834 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:55.194880 10698 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33611
I20260929 11:21:55.194902 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33611 every 8 connection(s)
I20260929 11:21:55.196269 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:21:55.198117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10698
I20260929 11:21:55.198385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:21:55.199030 388 external_mini_cluster.cc:1428] Setting key d288b1d017a4d3132cb92d1a061eebd2
I20260929 11:21:55.200058 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:55.211136 10765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193: Bootstrap starting.
Sep 29 11:21:55 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:55.214470 10765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:55.215474 10765 log.cc:824] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:55.218161 10765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193: No bootstrap required, opened a new log
I20260929 11:21:55.219250 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:55.200913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.0.97.62:33611)
0929 11:21:55.201704 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:55.201715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:55.201753 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:21:55.203143 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:55.203162 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:55.203531 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:55.203914 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:55.205391 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:55.206558 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:55.207675 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:55.208336 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:55.208513 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:55.212480 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:55.212489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:55.212504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:55.212545 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:55.215453 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:55.216083 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:55.216089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:55.216094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:55.216234 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:55.216703 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:55.216708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:55.216711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:55.217209 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:55.217519 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:55.217929 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 11:21:55.218346 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":415,"threads_started":1}
I20260929 11:21:55.226039 10765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2366871c047c28665415113ff6193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } }
I20260929 11:21:55.226325 10765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:55.226413 10765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f2366871c047c28665415113ff6193, State: Initialized, Role: FOLLOWER
I20260929 11:21:55.226888 10765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2366871c047c28665415113ff6193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } }
I20260929 11:21:55.227092 10765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:21:55.227233 10765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:21:55.227415 10765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:55.229609 10765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2366871c047c28665415113ff6193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } }
I20260929 11:21:55.230089 10765 leader_election.cc:302] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f2366871c047c28665415113ff6193; no voters:
I20260929 11:21:55.230705 10765 leader_election.cc:288] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:21:55.230803 10770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:55.231843 10770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [term 1 LEADER]: Becoming Leader. State: Replica: 22f2366871c047c28665415113ff6193, State: Running, Role: LEADER
I20260929 11:21:55.232347 10770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2366871c047c28665415113ff6193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } }
I20260929 11:21:55.232877 10765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:21:55.235006 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f2366871c047c28665415113ff6193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2366871c047c28665415113ff6193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } }
I20260929 11:21:55.235397 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:55.235248 10772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f2366871c047c28665415113ff6193. Latest consensus state: current_term: 1 leader_uuid: "22f2366871c047c28665415113ff6193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2366871c047c28665415113ff6193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } }
I20260929 11:21:55.235545 10772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193 [sys.catalog]: This master's current role is: LEADER
I20260929 11:21:55.237659 10778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:21:55.241981 10778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:21:55.249857 10778 catalog_manager.cc:1347] Generated new cluster ID: df399a07c025442281e1dc834cc80112
I20260929 11:21:55.249951 10778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:21:55.260254 10778 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:21:55.261698 10778 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:21:55.266809 10778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193: Generated new TSK 0
I20260929 11:21:55.267758 10778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:21:55.327584 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33611
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:55.646005 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:55.646283 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:55.646451 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:55.653798 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:55.653906 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:55.654094 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:21:55.663921 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:55.665302 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:55.666574 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:55.679033 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:55.679195 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:55.679121 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:55.679975 10791 server_base.cc:1034] running on GCE node
I20260929 11:21:55.680526 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:55.681458 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:55.682657 10791 hybrid_clock.cc:648] HybridClock initialized: now 1790680915682608 us; error 34 us; skew 500 ppm
I20260929 11:21:55.682932 10791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:55 dist-test-slave-srst krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:55.686185 10791 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:21:55.687522 10791 webserver.cc:460] Webserver started at http://127.0.97.1:34135/ using document root <none> and password file <none>
I20260929 11:21:55.688187 10791 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:55.688287 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:55.688522 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:55.691358 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "a9fcce48ee3a4fe0ada4b725809be7db"
format_stamp: "Formatted at 2026-09-29 11:21:55 on dist-test-slave-srst"
server_key: "52de4fa85d2f5a964121e07872be4b23"
server_key_iv: "17d3e382515e6aa82542d26e50dcb309"
server_key_version: "encryptionkey@0"
I20260929 11:21:55.692022 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a9fcce48ee3a4fe0ada4b725809be7db"
format_stamp: "Formatted at 2026-09-29 11:21:55 on dist-test-slave-srst"
server_key: "52de4fa85d2f5a964121e07872be4b23"
server_key_iv: "17d3e382515e6aa82542d26e50dcb309"
server_key_version: "encryptionkey@0"
I20260929 11:21:55.696907 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 11:21:55.700101 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:55.701418 10791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:21:55.701545 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "a9fcce48ee3a4fe0ada4b725809be7db"
format_stamp: "Formatted at 2026-09-29 11:21:55 on dist-test-slave-srst"
server_key: "52de4fa85d2f5a964121e07872be4b23"
server_key_iv: "17d3e382515e6aa82542d26e50dcb309"
server_key_version: "encryptionkey@0"
I20260929 11:21:55.701692 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:55.713644 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:55.718003 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:55.718258 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:55.719137 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:55.720393 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:55.720495 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:55.720670 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:55.720750 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 11:21:55.752389 10791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41129
I20260929 11:21:55.752434 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41129 every 8 connection(s)
I20260929 11:21:55.753813 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:21:55.758615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10791
I20260929 11:21:55.758844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:21:55.759414 388 external_mini_cluster.cc:1428] Setting key 78f46582770570bc6b0bca5258946109
Sep 29 11:21:55 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:55.773793 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:55.756281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40131 (local address 127.0.97.62:33611)
0929 11:21:55.756732 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:55.756737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:55.757792 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 11:21:55.759835 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:55.759840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:55.759926 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:55.760040 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:55.761544 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:55.762238 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:55.763781 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:55.764182 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:55.764336 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:55.768130 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:55.768135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:55.768138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:55.768168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:55.770723 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:55.771454 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:55.771464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:55.771466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:55.771575 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:55.771997 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:55.772002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:55.772005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:55.772281 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:55.772429 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:55.773182 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 11:21:55.773416 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260929 11:21:55.774804 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:55.756808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33611 (local address 127.0.97.1:40131)
0929 11:21:55.757617 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:55.757675 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:55.759581 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:55.760281 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:55.760289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:55.760836 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:55.761389 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:55.761404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:55.762398 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:55.762409 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:55.763636 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:55.763647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:55.764438 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:55.764449 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:55.764729 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:55.765425 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:55.765446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:55.767884 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:55.770887 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:55.770896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:55.770914 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:55.771327 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:55.771685 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:55.771690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:55.771693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:55.771882 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:55.772403 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:55.772411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:55.772835 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 11:21:55.773096 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 11:21:55.773627 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":344,"threads_started":1}
I20260929 11:21:55.776459 10919 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33611
I20260929 11:21:55.776775 10919 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:55.777496 10919 heartbeater.cc:511] Master 127.0.97.62:33611 requested a full tablet report, sending...
I20260929 11:21:55.779482 10730 ts_manager.cc:194] Registered new tserver with Master: a9fcce48ee3a4fe0ada4b725809be7db (127.0.97.1:41129)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:21:55.780954 10730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:21:55.808805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33611
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:56.124040 10925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:56.124317 10925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:56.124430 10925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:56.131628 10925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:56.131728 10925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:56.131837 10925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:21:56.141482 10925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:56.142721 10925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:56.143987 10925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:56.156757 10933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:56.156862 10930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:56.156854 10931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:56.162057 10925 server_base.cc:1034] running on GCE node
I20260929 11:21:56.162688 10925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:56.163625 10925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:56.164835 10925 hybrid_clock.cc:648] HybridClock initialized: now 1790680916164790 us; error 33 us; skew 500 ppm
I20260929 11:21:56.165153 10925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:56 dist-test-slave-srst krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:56.168067 10925 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:21:56.169457 10925 webserver.cc:460] Webserver started at http://127.0.97.2:43859/ using document root <none> and password file <none>
I20260929 11:21:56.170137 10925 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:56.170275 10925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:56.170568 10925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:56.173458 10925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "c7f5643c8e384116b6b17d0ed523d3e9"
format_stamp: "Formatted at 2026-09-29 11:21:56 on dist-test-slave-srst"
server_key: "1b4f870a3969f841a71fed920919f763"
server_key_iv: "b218ed846c19fff270bfd3ffdc5ca0d0"
server_key_version: "encryptionkey@0"
I20260929 11:21:56.174232 10925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c7f5643c8e384116b6b17d0ed523d3e9"
format_stamp: "Formatted at 2026-09-29 11:21:56 on dist-test-slave-srst"
server_key: "1b4f870a3969f841a71fed920919f763"
server_key_iv: "b218ed846c19fff270bfd3ffdc5ca0d0"
server_key_version: "encryptionkey@0"
I20260929 11:21:56.179157 10925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:21:56.182307 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:56.183584 10925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:21:56.183744 10925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "c7f5643c8e384116b6b17d0ed523d3e9"
format_stamp: "Formatted at 2026-09-29 11:21:56 on dist-test-slave-srst"
server_key: "1b4f870a3969f841a71fed920919f763"
server_key_iv: "b218ed846c19fff270bfd3ffdc5ca0d0"
server_key_version: "encryptionkey@0"
I20260929 11:21:56.183954 10925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:56.205616 10925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:56.209537 10925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:56.209811 10925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:56.210672 10925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:56.211867 10925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:56.211966 10925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:56.212101 10925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:56.212169 10925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:56.244146 10925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43357
I20260929 11:21:56.244204 11052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43357 every 8 connection(s)
I20260929 11:21:56.245488 10925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:21:56.249603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 10925
I20260929 11:21:56.249800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:21:56.250353 388 external_mini_cluster.cc:1428] Setting key 3165ad201343d26b8d35c7b82333dd49
Sep 29 11:21:56 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:56.269317 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.248035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42075 (local address 127.0.97.62:33611)
0929 11:21:56.248258 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.248264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.249571 (+ 1307us) server_negotiation.cc:407] Connection header received
0929 11:21:56.251901 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.251908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.252016 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.252119 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:56.254328 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.255050 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.257638 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.258342 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.258492 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.262343 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:56.262352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:56.262361 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:56.262406 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:56.265980 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:56.266694 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:56.266703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:56.266709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:56.266865 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:56.267376 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:56.267383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:56.267390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:56.267756 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:56.267905 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.268584 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.268860 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:21:56.270061 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.248585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33611 (local address 127.0.97.2:42075)
0929 11:21:56.249371 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.249440 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.251625 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.252452 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.252465 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.253265 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:56.254122 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.254157 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.255218 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.255225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.257426 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.257443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.258598 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.258606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.258855 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.259581 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:56.259607 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:56.262101 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:56.266133 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:56.266140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:56.266154 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:56.266561 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:56.266988 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:56.266994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:56.266997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:56.267257 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:56.267883 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:56.267888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:56.268193 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.268438 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.268908 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":254,"threads_started":1}
I20260929 11:21:56.271502 11053 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33611
I20260929 11:21:56.271807 11053 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:56.272501 11053 heartbeater.cc:511] Master 127.0.97.62:33611 requested a full tablet report, sending...
I20260929 11:21:56.273974 10730 ts_manager.cc:194] Registered new tserver with Master: c7f5643c8e384116b6b17d0ed523d3e9 (127.0.97.2:43357)
I20260929 11:21:56.274896 10730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42075
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:21:56.303717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33611
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:21:56.621297 11058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:21:56.621558 11058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:21:56.621724 11058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:21:56.628660 11058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:21:56.628811 11058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:21:56.629026 11058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:21:56.638489 11058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:21:56.639739 11058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:21:56.641072 11058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:21:56.653827 11066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:56.653791 11064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:21:56.653862 11063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:21:56.654546 11058 server_base.cc:1034] running on GCE node
I20260929 11:21:56.655160 11058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:21:56.656067 11058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:21:56.657282 11058 hybrid_clock.cc:648] HybridClock initialized: now 1790680916657243 us; error 49 us; skew 500 ppm
I20260929 11:21:56.657636 11058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:21:56 dist-test-slave-srst krb5kdc[10685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:21:56.660697 11058 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:21:56.662135 11058 webserver.cc:460] Webserver started at http://127.0.97.3:45399/ using document root <none> and password file <none>
I20260929 11:21:56.662819 11058 fs_manager.cc:359] Metadata directory not provided
I20260929 11:21:56.662956 11058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:21:56.663237 11058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:21:56.666170 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "e949e05b82274cfaba9f9b3413439bda"
format_stamp: "Formatted at 2026-09-29 11:21:56 on dist-test-slave-srst"
server_key: "eaf27300f8fcc9d3ba27ea731c58140d"
server_key_iv: "5f19e6513a6ed7c105ca79ce95f882d6"
server_key_version: "encryptionkey@0"
I20260929 11:21:56.666916 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e949e05b82274cfaba9f9b3413439bda"
format_stamp: "Formatted at 2026-09-29 11:21:56 on dist-test-slave-srst"
server_key: "eaf27300f8fcc9d3ba27ea731c58140d"
server_key_iv: "5f19e6513a6ed7c105ca79ce95f882d6"
server_key_version: "encryptionkey@0"
I20260929 11:21:56.671943 11058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 11:21:56.675328 11073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:56.676674 11058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:21:56.676918 11058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "e949e05b82274cfaba9f9b3413439bda"
format_stamp: "Formatted at 2026-09-29 11:21:56 on dist-test-slave-srst"
server_key: "eaf27300f8fcc9d3ba27ea731c58140d"
server_key_iv: "5f19e6513a6ed7c105ca79ce95f882d6"
server_key_version: "encryptionkey@0"
I20260929 11:21:56.677147 11058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:21:56.700153 11058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:21:56.704246 11058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:21:56.704555 11058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:21:56.705561 11058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:21:56.706873 11058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:21:56.706946 11058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:56.707033 11058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:21:56.707073 11058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:21:56.739230 11058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34739
I20260929 11:21:56.739318 11185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34739 every 8 connection(s)
I20260929 11:21:56.740550 11058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:21:56.745411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 11058
I20260929 11:21:56.745609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:21:56.746021 388 external_mini_cluster.cc:1428] Setting key c0d8592ad2d6e3f9900dc05936723e27
Sep 29 11:21:56 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:21:56.760433 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.743083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42743 (local address 127.0.97.62:33611)
0929 11:21:56.743283 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.743289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.744580 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 11:21:56.746604 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.746608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.746671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.746789 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:56.748507 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.749283 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.750754 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.751138 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.751273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.754913 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:56.754917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:56.754920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:56.754951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:56.757511 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:56.758209 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:56.758213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:56.758215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:56.758335 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:56.758665 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:56.758668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:56.758670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:56.758915 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:56.759015 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.759868 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.760084 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:21:56.761513 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.743651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33611 (local address 127.0.97.3:42743)
0929 11:21:56.744371 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.744462 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.746297 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.747093 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.747106 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.747785 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:21:56.748329 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.748345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.749428 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.749434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.750619 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.750630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.751351 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.751359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.751590 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.752243 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:21:56.752262 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:21:56.754723 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:21:56.757639 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:56.757645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:56.757657 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:56.758116 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:56.758424 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:21:56.758426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:21:56.758428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:21:56.758582 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:21:56.759036 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:21:56.759043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:21:56.759454 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.759735 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.760291 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":285,"threads_started":1}
I20260929 11:21:56.762660 11186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33611
I20260929 11:21:56.762930 11186 heartbeater.cc:464] Registering TS with master...
I20260929 11:21:56.763546 11186 heartbeater.cc:511] Master 127.0.97.62:33611 requested a full tablet report, sending...
I20260929 11:21:56.764866 10730 ts_manager.cc:194] Registered new tserver with Master: e949e05b82274cfaba9f9b3413439bda (127.0.97.3:34739)
I20260929 11:21:56.765645 10730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42743
I20260929 11:21:56.773135 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:21:56.783744 10919 heartbeater.cc:503] Master 127.0.97.62:33611 was elected leader, sending a full tablet report...
I20260929 11:21:56.789862 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.777541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.0.97.62:33611)
0929 11:21:56.777704 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.777708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.778218 (+ 510us) server_negotiation.cc:407] Connection header received
0929 11:21:56.778363 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.778366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.778437 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.778512 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:56.779763 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.780489 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.781727 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.782099 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.782184 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.784540 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:56.784546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:56.784549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:56.784582 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:56.787643 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:56.788287 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:56.788291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:56.788293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:56.788423 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:56.788788 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:56.788790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:56.788792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:56.789060 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:56.789222 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.789508 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.789656 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:21:56.793432 10730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:21:56.799938 10730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:21:56.831297 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.811411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43357 (local address 127.0.0.1:52930)
0929 11:21:56.824401 (+ 12990us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.824415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.824569 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.825478 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.825483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.825508 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:56.825810 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.825818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.827081 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.827088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.828506 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.828518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.830369 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.830374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.830576 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.830644 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.830779 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.830969 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12885,"thread_start_us":12760,"threads_started":1}
I20260929 11:21:56.831322 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.811843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.0.97.2:43357)
0929 11:21:56.821806 (+ 9963us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.821818 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.824535 (+ 2717us) server_negotiation.cc:407] Connection header received
0929 11:21:56.824770 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.824780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.825063 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.825251 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:56.825969 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.826956 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.828667 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.830177 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.830309 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.830515 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.830832 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.830993 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9759,"thread_start_us":9618,"threads_started":1}
I20260929 11:21:56.835527 10988 tablet_service.cc:1467] Processing CreateTablet for tablet 078ba35fabfa45769628c238a1718f5f (DEFAULT_TABLE table=test-table [id=d73713c2122b4bea887d0f2f70e483e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:56.836213 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.811763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34739 (local address 127.0.0.1:41418)
0929 11:21:56.829105 (+ 17342us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.829123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.829243 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.829850 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.829854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.829878 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:56.830229 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.830238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.832134 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.832139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.834032 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.834048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.835416 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.835422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.835587 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.835629 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.835725 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.835842 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17189,"mutex_wait_us":1,"thread_start_us":11135,"threads_started":1}
I20260929 11:21:56.836259 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.812113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.0.97.3:34739)
0929 11:21:56.825066 (+ 12953us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.825074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.829253 (+ 4179us) server_negotiation.cc:407] Connection header received
0929 11:21:56.829382 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.829394 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.829656 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.829864 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:56.830459 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.832001 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.834199 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.835217 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.835352 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.835577 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.835772 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.835928 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12786,"thread_start_us":12960,"threads_started":1}
I20260929 11:21:56.837774 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.811269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.0.1:47204)
0929 11:21:56.823342 (+ 12073us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.823397 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.823601 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.830530 (+ 6929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.830537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.830560 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:56.830922 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.830934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.832823 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.832828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.834973 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.834987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.837060 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.837066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.837293 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.837323 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.837416 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.837533 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11769,"mutex_wait_us":16,"thread_start_us":17528,"threads_started":1}
I20260929 11:21:56.838133 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.811345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.0.97.1:41129)
0929 11:21:56.829778 (+ 18433us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.829787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.829813 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:21:56.829897 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.829905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.830139 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.830311 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:56.831131 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.832664 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.835145 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.836530 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.836698 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.836913 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.837484 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.837652 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18269,"thread_start_us":18124,"threads_started":1}
I20260929 11:21:56.838475 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ba35fabfa45769628c238a1718f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:56.839881 11121 tablet_service.cc:1467] Processing CreateTablet for tablet 078ba35fabfa45769628c238a1718f5f (DEFAULT_TABLE table=test-table [id=d73713c2122b4bea887d0f2f70e483e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:56.841192 11121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ba35fabfa45769628c238a1718f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:56.843708 10854 tablet_service.cc:1467] Processing CreateTablet for tablet 078ba35fabfa45769628c238a1718f5f (DEFAULT_TABLE table=test-table [id=d73713c2122b4bea887d0f2f70e483e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:21:56.845062 10854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078ba35fabfa45769628c238a1718f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:21:56.851785 11204 tablet_bootstrap.cc:492] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Bootstrap starting.
I20260929 11:21:56.853776 11205 tablet_bootstrap.cc:492] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Bootstrap starting.
I20260929 11:21:56.856804 11204 tablet_bootstrap.cc:654] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:56.857271 11205 tablet_bootstrap.cc:654] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:56.858251 11204 log.cc:824] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:56.858291 11206 tablet_bootstrap.cc:492] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Bootstrap starting.
I20260929 11:21:56.858505 11205 log.cc:824] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:56.861254 11204 tablet_bootstrap.cc:492] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: No bootstrap required, opened a new log
I20260929 11:21:56.861500 11204 ts_tablet_manager.cc:1397] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 11:21:56.861555 11205 tablet_bootstrap.cc:492] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: No bootstrap required, opened a new log
I20260929 11:21:56.861794 11205 ts_tablet_manager.cc:1397] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 11:21:56.861905 11206 tablet_bootstrap.cc:654] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Neither blocks nor log segments found. Creating new log.
I20260929 11:21:56.863080 11206 log.cc:824] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Log is configured to *not* fsync() on all Append() calls
I20260929 11:21:56.866336 11206 tablet_bootstrap.cc:492] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: No bootstrap required, opened a new log
I20260929 11:21:56.866602 11206 ts_tablet_manager.cc:1397] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 11:21:56.869310 11204 raft_consensus.cc:348] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.869653 11204 raft_consensus.cc:374] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:56.869764 11204 raft_consensus.cc:729] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e949e05b82274cfaba9f9b3413439bda, State: Initialized, Role: FOLLOWER
I20260929 11:21:56.870371 11204 consensus_queue.cc:260] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.871398 11186 heartbeater.cc:503] Master 127.0.97.62:33611 was elected leader, sending a full tablet report...
I20260929 11:21:56.872080 11204 ts_tablet_manager.cc:1428] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 11:21:56.871984 11205 raft_consensus.cc:348] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.872378 11205 raft_consensus.cc:374] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:56.872468 11205 raft_consensus.cc:729] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f5643c8e384116b6b17d0ed523d3e9, State: Initialized, Role: FOLLOWER
I20260929 11:21:56.873055 11205 consensus_queue.cc:260] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.874116 11053 heartbeater.cc:503] Master 127.0.97.62:33611 was elected leader, sending a full tablet report...
I20260929 11:21:56.874697 11205 ts_tablet_manager.cc:1428] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 11:21:56.876796 11206 raft_consensus.cc:348] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.877192 11206 raft_consensus.cc:374] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:21:56.877271 11206 raft_consensus.cc:729] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9fcce48ee3a4fe0ada4b725809be7db, State: Initialized, Role: FOLLOWER
I20260929 11:21:56.877774 11206 consensus_queue.cc:260] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.879321 11206 ts_tablet_manager.cc:1428] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 11:21:56.936817 11211 raft_consensus.cc:482] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:21:56.937160 11211 raft_consensus.cc:504] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.938930 11211 leader_election.cc:288] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9fcce48ee3a4fe0ada4b725809be7db (127.0.97.1:41129), e949e05b82274cfaba9f9b3413439bda (127.0.97.3:34739)
I20260929 11:21:56.948087 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.939388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39675 (local address 127.0.97.1:41129)
0929 11:21:56.939565 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.939571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.940442 (+ 871us) server_negotiation.cc:407] Connection header received
0929 11:21:56.940535 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.940541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.940641 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.940728 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:56.941524 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.942536 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.945566 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.946895 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.947043 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.947177 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.947669 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.947807 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:21:56.948083 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.939383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56253 (local address 127.0.97.3:34739)
0929 11:21:56.939565 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:56.939570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:56.940362 (+ 792us) server_negotiation.cc:407] Connection header received
0929 11:21:56.940545 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:56.940550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:56.940641 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:56.940728 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:21:56.941501 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.942545 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.945566 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.946897 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.947043 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.947177 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:56.947669 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 11:21:56.947807 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:21:56.948397 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.939257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.97.2:39675)
0929 11:21:56.940326 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.940345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.940432 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.940773 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.940777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.940861 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:56.941286 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.941303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.942664 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.942668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.945360 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.945376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.947188 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.947205 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.947467 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.947521 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.947612 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.947778 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":304,"spinlock_wait_cycles":22400,"thread_start_us":489,"threads_started":1}
I20260929 11:21:56.948513 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:56.939194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34739 (local address 127.0.97.2:56253)
0929 11:21:56.940241 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0929 11:21:56.940260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:21:56.940411 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:21:56.940773 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:21:56.940777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:21:56.940804 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:21:56.941307 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.941315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.942664 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.942668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.945360 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:21:56.945376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:21:56.947181 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:21:56.947187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:21:56.947492 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:56.947521 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:21:56.947612 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:21:56.947778 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":137,"thread_start_us":603,"threads_started":1}
I20260929 11:21:56.949525 11141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ba35fabfa45769628c238a1718f5f" candidate_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e949e05b82274cfaba9f9b3413439bda" is_pre_election: true
I20260929 11:21:56.950011 11141 raft_consensus.cc:2454] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7f5643c8e384116b6b17d0ed523d3e9 in term 0.
I20260929 11:21:56.950605 10943 leader_election.cc:302] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7f5643c8e384116b6b17d0ed523d3e9, e949e05b82274cfaba9f9b3413439bda; no voters:
I20260929 11:21:56.950708 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ba35fabfa45769628c238a1718f5f" candidate_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" is_pre_election: true
I20260929 11:21:56.950990 11211 raft_consensus.cc:2781] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:21:56.951131 11211 raft_consensus.cc:482] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:21:56.951241 11211 raft_consensus.cc:3037] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:56.951332 10874 raft_consensus.cc:2454] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7f5643c8e384116b6b17d0ed523d3e9 in term 0.
I20260929 11:21:56.953779 11211 raft_consensus.cc:504] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.954435 11211 leader_election.cc:288] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [CANDIDATE]: Term 1 election: Requested vote from peers a9fcce48ee3a4fe0ada4b725809be7db (127.0.97.1:41129), e949e05b82274cfaba9f9b3413439bda (127.0.97.3:34739)
I20260929 11:21:56.954823 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ba35fabfa45769628c238a1718f5f" candidate_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9fcce48ee3a4fe0ada4b725809be7db"
I20260929 11:21:56.954962 11141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078ba35fabfa45769628c238a1718f5f" candidate_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e949e05b82274cfaba9f9b3413439bda"
I20260929 11:21:56.955030 10874 raft_consensus.cc:3037] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:56.955175 11141 raft_consensus.cc:3037] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:21:56.958403 10874 raft_consensus.cc:2454] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7f5643c8e384116b6b17d0ed523d3e9 in term 1.
I20260929 11:21:56.958402 11141 raft_consensus.cc:2454] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7f5643c8e384116b6b17d0ed523d3e9 in term 1.
I20260929 11:21:56.958803 10941 leader_election.cc:302] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9fcce48ee3a4fe0ada4b725809be7db, c7f5643c8e384116b6b17d0ed523d3e9; no voters:
I20260929 11:21:56.959136 11211 raft_consensus.cc:2781] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:21:56.959635 11211 raft_consensus.cc:686] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 1 LEADER]: Becoming Leader. State: Replica: c7f5643c8e384116b6b17d0ed523d3e9, State: Running, Role: LEADER
I20260929 11:21:56.960183 11211 consensus_queue.cc:237] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } }
I20260929 11:21:56.964798 10730 catalog_manager.cc:5495] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7f5643c8e384116b6b17d0ed523d3e9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7f5643c8e384116b6b17d0ed523d3e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 } health_report { overall_health: UNKNOWN } } }
W20260929 11:21:56.992233 11187 tablet.cc:2369] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:21:56.997402 11054 tablet.cc:2369] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:21:57.006119 10920 tablet.cc:2369] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:21:57.015933 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:57.010828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.0.97.2:43357)
0929 11:21:57.011028 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:57.011034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:57.011124 (+ 90us) server_negotiation.cc:407] Connection header received
0929 11:21:57.011347 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:57.011351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:57.011419 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:57.011556 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:21:57.011906 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.012567 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.013883 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.014257 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.014393 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:57.014891 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:21:57.015357 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:21:57.015521 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:57.015595 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 11:21:57.015671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:21:57.023764 10874 raft_consensus.cc:1261] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 1 FOLLOWER]: Refusing update from remote peer c7f5643c8e384116b6b17d0ed523d3e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:57.023765 11141 raft_consensus.cc:1261] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 1 FOLLOWER]: Refusing update from remote peer c7f5643c8e384116b6b17d0ed523d3e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:21:57.024725 11215 consensus_queue.cc:1035] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e949e05b82274cfaba9f9b3413439bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:57.025100 11211 consensus_queue.cc:1035] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9fcce48ee3a4fe0ada4b725809be7db" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:21:57.034978 11218 mvcc.cc:205] Tried to move back new op lower bound from 7334629036119867392 to 7334629035873878016. Current Snapshot: MvccSnapshot[applied={T|T < 7334629036119867392}]
I20260929 11:21:57.035784 11220 mvcc.cc:205] Tried to move back new op lower bound from 7334629036119867392 to 7334629035873878016. Current Snapshot: MvccSnapshot[applied={T|T < 7334629036119867392}]
I20260929 11:21:57.047947 10730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:57.048211 10730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:21:57.051577 10730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22f2366871c047c28665415113ff6193: Sending DeleteTablet for 3 replicas of tablet 078ba35fabfa45769628c238a1718f5f
I20260929 11:21:57.052577 10854 tablet_service.cc:1514] Processing DeleteTablet for tablet 078ba35fabfa45769628c238a1718f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47204
I20260929 11:21:57.052632 11121 tablet_service.cc:1514] Processing DeleteTablet for tablet 078ba35fabfa45769628c238a1718f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41418
I20260929 11:21:57.052640 10988 tablet_service.cc:1514] Processing DeleteTablet for tablet 078ba35fabfa45769628c238a1718f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:21:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52930
I20260929 11:21:57.055028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10791
I20260929 11:21:57.068228 11227 tablet_replica.cc:330] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: stopping tablet replica
I20260929 11:21:57.068758 11227 raft_consensus.cc:2229] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:57.069326 11227 raft_consensus.cc:2258] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:57.072042 11227 ts_tablet_manager.cc:1905] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:57.073535 11228 tablet_replica.cc:330] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: stopping tablet replica
I20260929 11:21:57.073592 11229 tablet_replica.cc:330] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: stopping tablet replica
I20260929 11:21:57.073921 11228 raft_consensus.cc:2229] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:21:57.074103 11229 raft_consensus.cc:2229] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:21:57.074352 11228 raft_consensus.cc:2258] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:57.074887 11229 raft_consensus.cc:2258] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:21:57.077610 11228 ts_tablet_manager.cc:1905] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:57.077613 11229 ts_tablet_manager.cc:1905] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:21:57.079381 11227 ts_tablet_manager.cc:1918] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:57.079546 11227 log.cc:1196] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/078ba35fabfa45769628c238a1718f5f
I20260929 11:21:57.080090 11227 ts_tablet_manager.cc:1939] T 078ba35fabfa45769628c238a1718f5f P a9fcce48ee3a4fe0ada4b725809be7db: Deleting consensus metadata
I20260929 11:21:57.082050 10714 catalog_manager.cc:4841] TS a9fcce48ee3a4fe0ada4b725809be7db (127.0.97.1:41129): tablet 078ba35fabfa45769628c238a1718f5f (table test-table [id=d73713c2122b4bea887d0f2f70e483e8]) successfully deleted
Sep 29 11:21:57 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:21:57.084643 11229 ts_tablet_manager.cc:1918] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:57.084803 11229 log.cc:1196] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/078ba35fabfa45769628c238a1718f5f
I20260929 11:21:57.085326 11229 ts_tablet_manager.cc:1939] T 078ba35fabfa45769628c238a1718f5f P c7f5643c8e384116b6b17d0ed523d3e9: Deleting consensus metadata
I20260929 11:21:57.086935 10717 catalog_manager.cc:4841] TS c7f5643c8e384116b6b17d0ed523d3e9 (127.0.97.2:43357): tablet 078ba35fabfa45769628c238a1718f5f (table test-table [id=d73713c2122b4bea887d0f2f70e483e8]) successfully deleted
I20260929 11:21:57.087605 11228 ts_tablet_manager.cc:1918] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:21:57.087787 11228 log.cc:1196] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/078ba35fabfa45769628c238a1718f5f
I20260929 11:21:57.088330 11228 ts_tablet_manager.cc:1939] T 078ba35fabfa45769628c238a1718f5f P e949e05b82274cfaba9f9b3413439bda: Deleting consensus metadata
I20260929 11:21:57.089923 10716 catalog_manager.cc:4841] TS e949e05b82274cfaba9f9b3413439bda (127.0.97.3:34739): tablet 078ba35fabfa45769628c238a1718f5f (table test-table [id=d73713c2122b4bea887d0f2f70e483e8]) successfully deleted
I20260929 11:21:57.092763 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:57.056720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.97.1:41129)
0929 11:21:57.056948 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:57.056957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:57.068424 (+ 11467us) server_negotiation.cc:407] Connection header received
0929 11:21:57.068607 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:57.068612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:57.068721 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:57.068832 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:57.070215 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.071357 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.074716 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.075357 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.075480 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:57.085835 (+ 10355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:57.085848 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:57.085866 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:57.085903 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:57.090071 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:57.090723 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:57.090739 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:57.090741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:57.090860 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:57.091258 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:57.091266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:57.091268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:57.091611 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:57.091903 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:57.092206 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 11:21:57.092401 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:21:57.093400 10914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:57.197706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10791
I20260929 11:21:57.221282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10925
Sep 29 11:21:57 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:21:57.240173 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:57.222207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.97.2:43357)
0929 11:21:57.222413 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:57.222420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:57.222470 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:21:57.222699 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:57.222705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:57.222814 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:57.222931 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:57.224479 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.225866 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.227626 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.228343 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.228480 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:57.233554 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:57.233568 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:57.233589 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:57.233647 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:57.236963 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:57.237813 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:57.237821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:57.237824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:57.237982 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:57.238422 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:57.238429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:57.238435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:57.238874 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:57.239219 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:57.239514 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 11:21:57.239744 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:21:57.240993 11048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:57.350129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10925
I20260929 11:21:57.369277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 11058
Sep 29 11:21:57 dist-test-slave-srst krb5kdc[10685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:21:57.382867 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:21:57.369967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.97.3:34739)
0929 11:21:57.370142 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:21:57.370146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:21:57.370178 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:21:57.370322 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:21:57.370326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:21:57.370393 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:21:57.370485 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:21:57.371567 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.372208 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.373483 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:21:57.373841 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:21:57.373930 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:21:57.377993 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:21:57.378005 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:21:57.378021 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:21:57.378051 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:21:57.380518 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:57.381174 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:57.381181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:57.381183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:57.381336 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:21:57.381672 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:21:57.381675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:21:57.381677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:21:57.382083 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:21:57.382279 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 11:21:57.382491 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 11:21:57.382628 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:21:57.383471 11181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:57.469944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 11058
I20260929 11:21:57.487380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 10698
I20260929 11:21:57.488457 10760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:21:57.597433 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 10698
2026-09-29T11:21:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4091 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:21:57 dist-test-slave-srst krb5kdc[11237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:21:57 dist-test-slave-srst krb5kdc[11237](info): set up 2 sockets
Sep 29 11:21:57 dist-test-slave-srst krb5kdc[11237](info): commencing operation
krb5kdc: starting...
W20260929 11:21:58.665611 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:21:58 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:21:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 11:21:58.976939 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyNUK1V/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/postgres -l logfile start
2026-09-29 11:22:01.351 UTC [11267] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 11:22:01.351 UTC [11267] LOG: listening on IPv4 address "127.0.97.20", port 34217
2026-09-29 11:22:01.355 UTC [11267] LOG: listening on Unix socket "/tmp/.s.PGSQL.34217"
2026-09-29 11:22:01.376 UTC [11270] LOG: database system was shut down at 2026-09-29 11:22:00 UTC
2026-09-29 11:22:01.379 UTC [11267] LOG: database system is ready to accept connections
I20260929 11:22:02.333402 388 mini_postgres.cc:96] Postgres bound to 34217
2026-09-29 11:22:02.350 UTC [11278] FATAL: database "slave" does not exist
127.0.97.20:34217 - accepting connections
I20260929 11:22:02.351784 388 mini_ranger.cc:161] Starting Ranger...
I20260929 11:22:02.376701 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 11:22:02.820159 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 11:22:02.820303 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-admin
2026-09-29 11:22:03,100 [I] DB FLAVOR :POSTGRES
2026-09-29 11:22:03,101 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:22:03,101 [I] Checking connection..
2026-09-29 11:22:03,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 11:22:03,361 [I] Checking connection passed.
2026-09-29 11:22:03,361 [I] --------- Verifying version history table ---------
2026-09-29 11:22:03,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:22:03,620 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:22:03,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:22:03,863 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:22:03,863 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 11:22:03,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 11:22:04.096 UTC [11342] WARNING: there is no transaction in progress
2026-09-29 11:22:04,102 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 11:22:04,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:22:04,343 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 11:22:04,343 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:22:04,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 11:22:04,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 11:22:04,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srst.c.gcp-upstream.internal','N') ;"
2026-09-29 11:22:05,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:22:05,329 [I] Table x_portal_user does not exist in database ranger
2026-09-29 11:22:05,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:22:05,581 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 11:22:05,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:22:05,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:22:06,085 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 11:22:06,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 11:22:06.983 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:06.999 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.014 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.030 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.048 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.069 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.131 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.138 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.151 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.161 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.171 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.185 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.195 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.201 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.207 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07.214 UTC [11450] WARNING: there is no transaction in progress
2026-09-29 11:22:07,823 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 11:22:07,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 11:22:08,045 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 11:22:08,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:22:08,302 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 11:22:08,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:22:08,559 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 11:22:08,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:22:08,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:22:09,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srst.c.gcp-upstream.internal';"
2026-09-29 11:22:09,292 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 11:22:09,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 11:22:09,528 [I] DB_PATCHES have already been applied
I20260929 11:22:09.534238 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyNUK1V/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 11:22:09.534324 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 11:22:09.548732 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:41349
Sep 29, 2026 11:22:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:22:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:22:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 11:22:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 11:22:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 11:22:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 11:22:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 11:22:10.668268 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.317s user 0.007s sys 0.030s
Sep 29, 2026 11:22:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 11:22:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 11:22:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 11:22:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 11:22:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 11:22:31.242691 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 11:22:31 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 11:22:31.422868 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 11:22:31.443317 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 11:22:32.396023 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 11:22:32.396155 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms
I20260929 11:22:32.396291 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34217
2026-09-29 11:22:32,802 [I] DB FLAVOR :POSTGRES
2026-09-29 11:22:32,802 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:22:32,802 [I] Checking connection
2026-09-29 11:22:32,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 11:22:33,038 [I] connection success
2026-09-29 11:22:33,038 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 11:22:33,039 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 11:22:33,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 11:22:33,292 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 11:22:33,292 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:22:33,292 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 11:22:33,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 11:22:33,574 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 11:22:34.008978 388 mini_ranger_kms.cc:310] Created kms service
I20260929 11:22:34.174830 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 11:22:34.223107 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 11:22:34 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:22:34.637616 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 11:22:34.637995 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 11:22:34.638070 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 11:22:34.658710 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11630
I20260929 11:22:34.658907 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:50473
Sep 29, 2026 11:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 11:22:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 11:22:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 11:22:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 11:22:36 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790680956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:22:36.993507 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.571s user 0.107s sys 0.322s
I20260929 11:22:36.993748 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 11:22:36.993948 388 mini_ranger_kms.cc:401] 127.0.97.20:50473/kms/v1/keys
Sep 29 11:22:37 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 11:22:37 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 11:22:39 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:22:39.803930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42735
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42735
--ranger_config_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50473/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:22:40.135540 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:22:40.135815 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:22:40.135900 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:22:40.143000 11681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:22:40.143131 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:22:40.143210 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:22:40.143292 11681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:22:40.143352 11681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:22:40.153455 11681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:50473/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42735
--ranger_config_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:22:40.154733 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:22:40.156006 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:22:40.167842 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:40.167935 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:40.167951 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:22:40.169039 11681 server_base.cc:1034] running on GCE node
I20260929 11:22:40.169786 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:22:40.171185 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:22:40.172413 11681 hybrid_clock.cc:648] HybridClock initialized: now 1790680960172372 us; error 58 us; skew 500 ppm
I20260929 11:22:40.172830 11681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:22:40 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:22:40.179536 11681 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:22:40.181066 11681 webserver.cc:460] Webserver started at http://127.0.97.62:44149/ using document root <none> and password file <none>
I20260929 11:22:40.181716 11681 fs_manager.cc:359] Metadata directory not provided
I20260929 11:22:40.181856 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:22:40.182143 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:22:40 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:22:40.230732 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "34b92a7afc4e4b379a5b185422021120"
format_stamp: "Formatted at 2026-09-29 11:22:40 on dist-test-slave-srst"
server_key: "81e927f5cc00b3d37e47826a5e0e6f40"
server_key_iv: "445331df159a33d8fd002fc955e17abc"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:40.231603 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "34b92a7afc4e4b379a5b185422021120"
format_stamp: "Formatted at 2026-09-29 11:22:40 on dist-test-slave-srst"
server_key: "81e927f5cc00b3d37e47826a5e0e6f40"
server_key_iv: "445331df159a33d8fd002fc955e17abc"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:40.237005 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:22:40.256150 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:40.257718 11681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:22:40.257906 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "34b92a7afc4e4b379a5b185422021120"
format_stamp: "Formatted at 2026-09-29 11:22:40 on dist-test-slave-srst"
server_key: "81e927f5cc00b3d37e47826a5e0e6f40"
server_key_iv: "445331df159a33d8fd002fc955e17abc"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:40.258126 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:22:40.276727 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:22:40.280831 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:22:40.281123 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:22:40.302096 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42735
I20260929 11:22:40.302114 11747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42735 every 8 connection(s)
I20260929 11:22:40.303525 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:22:40.308205 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:22:40.309651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 11681
I20260929 11:22:40.309854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:22:40.310438 388 external_mini_cluster.cc:1428] Setting key abc30ddfe62a99f9546da8407424456a
I20260929 11:22:40.321619 11748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120: Bootstrap starting.
I20260929 11:22:40.325482 11748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120: Neither blocks nor log segments found. Creating new log.
I20260929 11:22:40.326552 11748 log.cc:824] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120: Log is configured to *not* fsync() on all Append() calls
I20260929 11:22:40.329540 11748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120: No bootstrap required, opened a new log
Sep 29 11:22:40 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:22:40.339313 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:40.312536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.97.62:42735)
0929 11:22:40.317199 (+ 4663us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:40.317219 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:40.317890 (+ 671us) server_negotiation.cc:407] Connection header received
0929 11:22:40.320715 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:40.320744 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:40.321260 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:40.321696 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:40.322834 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:40.324913 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:40.326220 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:40.327303 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:40.327434 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:40.331671 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:40.331678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:40.331692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:40.331732 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:40.335166 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:40.336006 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:40.336012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:40.336018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:40.336151 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:40.336527 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:40.336530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:40.336532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:40.337165 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:40.337385 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:40.337861 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 11:22:40.338266 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4393,"thread_start_us":4121,"threads_started":1}
I20260929 11:22:40.339206 11748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b92a7afc4e4b379a5b185422021120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } }
I20260929 11:22:40.339638 11748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:22:40.339740 11748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b92a7afc4e4b379a5b185422021120, State: Initialized, Role: FOLLOWER
I20260929 11:22:40.340346 11748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b92a7afc4e4b379a5b185422021120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } }
I20260929 11:22:40.340623 11748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:22:40.340783 11748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:22:40.340975 11748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:22:40.343194 11748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b92a7afc4e4b379a5b185422021120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } }
I20260929 11:22:40.343693 11748 leader_election.cc:302] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b92a7afc4e4b379a5b185422021120; no voters:
I20260929 11:22:40.344259 11748 leader_election.cc:288] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:22:40.344439 11753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:22:40.345196 11753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [term 1 LEADER]: Becoming Leader. State: Replica: 34b92a7afc4e4b379a5b185422021120, State: Running, Role: LEADER
I20260929 11:22:40.345773 11753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b92a7afc4e4b379a5b185422021120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } }
I20260929 11:22:40.346378 11748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:22:40.349056 11754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b92a7afc4e4b379a5b185422021120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b92a7afc4e4b379a5b185422021120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } }
I20260929 11:22:40.349488 11754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [sys.catalog]: This master's current role is: LEADER
I20260929 11:22:40.349960 11755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b92a7afc4e4b379a5b185422021120. Latest consensus state: current_term: 1 leader_uuid: "34b92a7afc4e4b379a5b185422021120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b92a7afc4e4b379a5b185422021120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } }
I20260929 11:22:40.350296 11755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120 [sys.catalog]: This master's current role is: LEADER
I20260929 11:22:40.351024 11758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:22:40.356484 11758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:22:40.358771 11748 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:22:40.376375 11758 catalog_manager.cc:1347] Generated new cluster ID: 4b34bea661f44fe2a759670c6c7b09b1
I20260929 11:22:40.376513 11758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:22:40.394768 11758 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:22:40.396849 11758 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:22:40.410069 11758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120: Generated new TSK 0
I20260929 11:22:40.411307 11758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:22:41.112628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42735
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:22:41 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790680961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 11:22:41.523676 11793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:22:41.524000 11793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:22:41.524180 11793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:22:41.532949 11793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:22:41.533171 11793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:22:41.533406 11793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:22:41.546332 11793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:50473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:22:41.548512 11793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:22:41.550392 11793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:22:41.569375 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:41.570290 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:41.578236 11806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:22:41.579175 11793 server_base.cc:1034] running on GCE node
I20260929 11:22:41.579871 11793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:22:41.580998 11793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:22:41.582280 11793 hybrid_clock.cc:648] HybridClock initialized: now 1790680961582211 us; error 62 us; skew 500 ppm
I20260929 11:22:41.582645 11793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:22:41 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:22:41.586041 11793 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:22:41.587931 11793 webserver.cc:460] Webserver started at http://127.0.97.1:39629/ using document root <none> and password file <none>
I20260929 11:22:41.588750 11793 fs_manager.cc:359] Metadata directory not provided
I20260929 11:22:41.588887 11793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:22:41.589205 11793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:22:41 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:22:41.615661 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "b3577c41621544ea8a94020eec21dbe9"
format_stamp: "Formatted at 2026-09-29 11:22:41 on dist-test-slave-srst"
server_key: "be8796c05ae85bf3634e5d837cf6e9e6"
server_key_iv: "508f722a8dbce1481c4678a01dd25668"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:41.617712 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b3577c41621544ea8a94020eec21dbe9"
format_stamp: "Formatted at 2026-09-29 11:22:41 on dist-test-slave-srst"
server_key: "be8796c05ae85bf3634e5d837cf6e9e6"
server_key_iv: "508f722a8dbce1481c4678a01dd25668"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:41.632480 11793 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.012s sys 0.000s
I20260929 11:22:41.656272 11813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:41.658303 11793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:22:41.658507 11793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "b3577c41621544ea8a94020eec21dbe9"
format_stamp: "Formatted at 2026-09-29 11:22:41 on dist-test-slave-srst"
server_key: "be8796c05ae85bf3634e5d837cf6e9e6"
server_key_iv: "508f722a8dbce1481c4678a01dd25668"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:41.658779 11793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:22:41.684297 11793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:22:41.690110 11793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:22:41.690549 11793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:22:41.697942 11793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:22:41.700233 11793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:22:41.700354 11793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:41.700490 11793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:22:41.700610 11793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 11:22:41.820760 11793 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43523
I20260929 11:22:41.820801 11925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43523 every 8 connection(s)
I20260929 11:22:41.822682 11793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:22:41.830940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 11793
I20260929 11:22:41.831133 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:22:41.831871 388 external_mini_cluster.cc:1428] Setting key 94adbcea70c271d9496477a956dcc3cc
Sep 29 11:22:41 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:22:41.857316 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:41.828190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40615 (local address 127.0.97.62:42735)
0929 11:22:41.833408 (+ 5218us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:41.833414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:41.833432 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:22:41.833517 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:41.833522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:41.833614 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:41.833708 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:41.837587 (+ 3879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:41.838682 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:41.840986 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:41.841656 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:41.841749 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:41.847143 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:41.847150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:41.847154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:41.847216 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:41.850880 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:41.854382 (+ 3502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:41.854388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:41.854392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:41.854544 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:41.855140 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:41.855145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:41.855148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:41.855496 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:41.855624 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:41.856829 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0929 11:22:41.857048 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5092,"thread_start_us":9022,"threads_started":1}
I20260929 11:22:41.859759 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:41.827539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42735 (local address 127.0.97.1:40615)
0929 11:22:41.829915 (+ 2376us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:41.829973 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:41.832244 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:41.835885 (+ 3641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:41.835897 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:41.836550 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:22:41.837367 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:41.837391 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:41.838878 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:41.838883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:41.840826 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:41.840842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:41.842096 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:41.842102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:41.842368 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:41.843272 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:22:41.843300 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:22:41.846888 (+ 3588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:22:41.852696 (+ 5808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:22:41.852707 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:22:41.852726 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:22:41.853391 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:22:41.854716 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:22:41.854720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:22:41.854724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:22:41.855008 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:22:41.855909 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:22:41.855917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:22:41.856354 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 11:22:41.857729 (+ 1375us) client_negotiation.cc:240] Negotiation successful
0929 11:22:41.858301 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2066,"thread_start_us":1825,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:22:41.870123 11926 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42735
I20260929 11:22:41.870806 11926 heartbeater.cc:464] Registering TS with master...
I20260929 11:22:41.871881 11926 heartbeater.cc:511] Master 127.0.97.62:42735 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 11:22:41.874812 11713 ts_manager.cc:194] Registered new tserver with Master: b3577c41621544ea8a94020eec21dbe9 (127.0.97.1:43523)
I20260929 11:22:41.876932 11713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40615
Sep 29 11:22:41 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:22:41.921586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42735
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:22:42 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 11:22:42.385340 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:22:42.385784 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:22:42.385968 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:22:42.397346 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:22:42.397557 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:22:42.397768 11932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:22:42.413758 11932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:50473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:22:42.415629 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:22:42.417342 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:22:42.434724 11939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:42.435349 11940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:22:42.444751 11932 server_base.cc:1034] running on GCE node
W20260929 11:22:42.445472 11942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:22:42.446298 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:22:42.447642 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:22:42.448894 11932 hybrid_clock.cc:648] HybridClock initialized: now 1790680962448800 us; error 95 us; skew 500 ppm
I20260929 11:22:42.449288 11932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:22:42 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:22:42.455992 11932 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:22:42.470378 11932 webserver.cc:460] Webserver started at http://127.0.97.2:34575/ using document root <none> and password file <none>
I20260929 11:22:42.471864 11932 fs_manager.cc:359] Metadata directory not provided
I20260929 11:22:42.472008 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:22:42.472360 11932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:22:42 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:22:42.489077 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "27dcebe0a40041ae86fe308a376484b9"
format_stamp: "Formatted at 2026-09-29 11:22:42 on dist-test-slave-srst"
server_key: "e640488e79e66cbccd4bacff06d1b6a1"
server_key_iv: "bd2edbe2619dc17399ff6a7920b67221"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:42.489904 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "27dcebe0a40041ae86fe308a376484b9"
format_stamp: "Formatted at 2026-09-29 11:22:42 on dist-test-slave-srst"
server_key: "e640488e79e66cbccd4bacff06d1b6a1"
server_key_iv: "bd2edbe2619dc17399ff6a7920b67221"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:42.494994 11932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:22:42.508387 11956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:42.510311 11932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:22:42.510499 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "27dcebe0a40041ae86fe308a376484b9"
format_stamp: "Formatted at 2026-09-29 11:22:42 on dist-test-slave-srst"
server_key: "e640488e79e66cbccd4bacff06d1b6a1"
server_key_iv: "bd2edbe2619dc17399ff6a7920b67221"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:42.510720 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:22:42.526263 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:22:42.530186 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:22:42.530469 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:22:42.531374 11932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:22:42.532855 11932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:22:42.532950 11932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:42.533099 11932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:22:42.533170 11932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:42.570513 11932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40713
I20260929 11:22:42.570957 12068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40713 every 8 connection(s)
I20260929 11:22:42.572234 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:22:42.578013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 11932
I20260929 11:22:42.578207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:22:42.578649 388 external_mini_cluster.cc:1428] Setting key cc6a62a453cc4696e76186d52cfb9c8b
Sep 29 11:22:42 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:22:42.593441 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:42.575082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43965 (local address 127.0.97.62:42735)
0929 11:22:42.575528 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:42.575534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:42.576663 (+ 1129us) server_negotiation.cc:407] Connection header received
0929 11:22:42.579102 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:42.579108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:42.579195 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:42.579316 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:42.580980 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:42.581658 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:42.583169 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:42.583558 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:42.583691 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:42.586927 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:42.586932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:42.586936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:42.586966 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:42.590023 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:42.590769 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:42.590774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:42.590778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:42.591303 (+ 525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:42.591733 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:42.591738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:42.591741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:42.592001 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:42.592155 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:42.592870 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 11:22:42.593076 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":211,"threads_started":1}
I20260929 11:22:42.594519 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:42.575648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42735 (local address 127.0.97.2:43965)
0929 11:22:42.576447 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:42.576521 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:42.578772 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:42.579545 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:42.579553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:42.580082 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:22:42.580829 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:42.580848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:42.581802 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:42.581807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:42.583022 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:42.583042 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:42.583780 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:42.583787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:42.584063 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:42.584749 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:22:42.584770 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:22:42.586719 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:22:42.590215 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:22:42.590223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:22:42.590239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:22:42.590651 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:22:42.591411 (+ 760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:22:42.591416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:22:42.591420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:22:42.591620 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:22:42.592129 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:22:42.592136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:22:42.592473 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 11:22:42.592774 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 11:22:42.593271 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":360,"threads_started":1}
I20260929 11:22:42.596158 12069 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42735
I20260929 11:22:42.596486 12069 heartbeater.cc:464] Registering TS with master...
I20260929 11:22:42.597306 12069 heartbeater.cc:511] Master 127.0.97.62:42735 requested a full tablet report, sending...
I20260929 11:22:42.598732 11713 ts_manager.cc:194] Registered new tserver with Master: 27dcebe0a40041ae86fe308a376484b9 (127.0.97.2:40713)
I20260929 11:22:42.599687 11713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43965
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:22:42.641004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50473/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42735
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:22:42.880991 11926 heartbeater.cc:503] Master 127.0.97.62:42735 was elected leader, sending a full tablet report...
W20260929 11:22:43.004699 12075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:22:43.004971 12075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:22:43.005127 12075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:22:43.012467 12075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:22:43.012566 12075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:22:43.012785 12075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:22:43.022495 12075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:50473/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:22:43.023837 12075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:22:43.025214 12075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:22:43.038662 12081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:43.038857 12080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:22:43.039251 12083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:22:43.039736 12075 server_base.cc:1034] running on GCE node
I20260929 11:22:43.040236 12075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:22:43.041132 12075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:22:43.042326 12075 hybrid_clock.cc:648] HybridClock initialized: now 1790680963042290 us; error 25 us; skew 500 ppm
I20260929 11:22:43.042620 12075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:22:43 dist-test-slave-srst krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:22:43.045643 12075 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:22:43.047101 12075 webserver.cc:460] Webserver started at http://127.0.97.3:34193/ using document root <none> and password file <none>
I20260929 11:22:43.047730 12075 fs_manager.cc:359] Metadata directory not provided
I20260929 11:22:43.047819 12075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:22:43.048050 12075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:22:43 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:22:43.063849 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "45892a0c4aee4f12b0c8f0e6e9606860"
format_stamp: "Formatted at 2026-09-29 11:22:43 on dist-test-slave-srst"
server_key: "2da90d79ee35a8c71ed26a4a098d75db"
server_key_iv: "7560edfd6e5f529cdf343afbd9a9c6e7"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:43.064854 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "45892a0c4aee4f12b0c8f0e6e9606860"
format_stamp: "Formatted at 2026-09-29 11:22:43 on dist-test-slave-srst"
server_key: "2da90d79ee35a8c71ed26a4a098d75db"
server_key_iv: "7560edfd6e5f529cdf343afbd9a9c6e7"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:43.070008 12075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:22:43.084049 12090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:43.085736 12075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:22:43.085928 12075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "45892a0c4aee4f12b0c8f0e6e9606860"
format_stamp: "Formatted at 2026-09-29 11:22:43 on dist-test-slave-srst"
server_key: "2da90d79ee35a8c71ed26a4a098d75db"
server_key_iv: "7560edfd6e5f529cdf343afbd9a9c6e7"
server_key_version: "kuduclusterkey@0"
I20260929 11:22:43.086189 12075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:22:43.116539 12075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:22:43.120502 12075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:22:43.120815 12075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:22:43.121709 12075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:22:43.123440 12075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:22:43.123548 12075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:43.123672 12075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:22:43.123752 12075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:22:43.156572 12075 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46349
I20260929 11:22:43.156610 12202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46349 every 8 connection(s)
I20260929 11:22:43.157960 12075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:22:43.167646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 12075
I20260929 11:22:43.167835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:22:43.168365 388 external_mini_cluster.cc:1428] Setting key 07832753c41f82ed34f8406023a75ff1
Sep 29 11:22:43 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:22:43.177330 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.160457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34723 (local address 127.0.97.62:42735)
0929 11:22:43.160975 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.160980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.162557 (+ 1577us) server_negotiation.cc:407] Connection header received
0929 11:22:43.164238 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.164242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.164318 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.164416 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:43.165970 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.166640 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.168153 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.168790 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.168921 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.172041 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:43.172064 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:43.172067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:43.172097 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:43.174819 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.175423 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.175426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.175428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.175541 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.175913 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.175916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.175918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.176165 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:43.176276 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.176905 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.177073 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
I20260929 11:22:43.178297 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.161087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42735 (local address 127.0.97.3:34723)
0929 11:22:43.162351 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:43.162407 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:43.163997 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:43.164720 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:43.164728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:43.165158 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:22:43.165832 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.165851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.166757 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.166760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.168021 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.168032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.169020 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.169027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.169270 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.169932 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:22:43.169956 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:22:43.171853 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:22:43.174934 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:22:43.174940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:22:43.174952 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:22:43.175329 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:22:43.175626 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:22:43.175629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:22:43.175631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:22:43.175827 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:22:43.176263 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:22:43.176268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:22:43.176536 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 11:22:43.176832 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 11:22:43.177196 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"thread_start_us":329,"threads_started":1}
I20260929 11:22:43.179694 12203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42735
I20260929 11:22:43.180016 12203 heartbeater.cc:464] Registering TS with master...
I20260929 11:22:43.180825 12203 heartbeater.cc:511] Master 127.0.97.62:42735 requested a full tablet report, sending...
I20260929 11:22:43.182292 11713 ts_manager.cc:194] Registered new tserver with Master: 45892a0c4aee4f12b0c8f0e6e9606860 (127.0.97.3:46349)
I20260929 11:22:43.183319 11713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34723
I20260929 11:22:43.183440 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:22:43.200654 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.188268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.97.62:42735)
0929 11:22:43.188410 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.188414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.188943 (+ 529us) server_negotiation.cc:407] Connection header received
0929 11:22:43.189161 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.189167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.189256 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.189354 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:43.190914 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.191595 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.192977 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.193394 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.193515 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.195825 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:43.195829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:43.195833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:43.195860 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:43.198148 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.198781 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.198787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.198791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.198919 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.199290 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.199297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.199300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.199556 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:43.199782 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.200097 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.200307 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:22:43.204466 11713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:22:43.211642 11713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:22:43.238624 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.223914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40713 (local address 127.0.0.1:50984)
0929 11:22:43.224746 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:43.224802 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:43.225007 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:43.229502 (+ 4495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:43.229514 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:43.229549 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:22:43.229987 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.230005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.233026 (+ 3021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.233031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.235556 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.235573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.237700 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.237706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.237924 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.237976 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 11:22:43.238085 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 11:22:43.238285 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":302,"threads_started":1}
I20260929 11:22:43.239920 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.224117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.0.97.2:40713)
0929 11:22:43.228379 (+ 4262us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.228386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.228409 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:22:43.228530 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.228540 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.229257 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.230214 (+ 957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:22:43.230457 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.232881 (+ 2424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.235741 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.237490 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.239184 (+ 1694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.239350 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.239415 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.239583 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4107,"thread_start_us":4000,"threads_started":1}
I20260929 11:22:43.240634 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.223914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43523 (local address 127.0.0.1:39036)
0929 11:22:43.228936 (+ 5022us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:43.229492 (+ 556us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:43.229592 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:43.230462 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:43.230466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:43.230490 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:22:43.230876 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.230887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.233868 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.233873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.237086 (+ 3213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.237100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.239444 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.239450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.239612 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.239640 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:22:43.239735 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 11:22:43.239851 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4895,"mutex_wait_us":517,"thread_start_us":4232,"threads_started":1}
I20260929 11:22:43.240830 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.224239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.0.97.1:43523)
0929 11:22:43.229437 (+ 5198us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.229447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.229725 (+ 278us) server_negotiation.cc:407] Connection header received
0929 11:22:43.229829 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.229837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.230060 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.230238 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:22:43.232095 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.233697 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.237263 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.238911 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.239042 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.239309 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.240254 (+ 945us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.240427 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5021,"thread_start_us":4917,"threads_started":1}
I20260929 11:22:43.243412 12004 tablet_service.cc:1467] Processing CreateTablet for tablet b5786922b2e4452e897ca74024a06ded (DEFAULT_TABLE table=test-table [id=67326c866c8a40fdbda7d2434f5fe840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:22:43.244072 11861 tablet_service.cc:1467] Processing CreateTablet for tablet b5786922b2e4452e897ca74024a06ded (DEFAULT_TABLE table=test-table [id=67326c866c8a40fdbda7d2434f5fe840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:22:43.244764 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5786922b2e4452e897ca74024a06ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:22:43.245358 11861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5786922b2e4452e897ca74024a06ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:22:43.249047 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.229047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46349 (local address 127.0.0.1:55924)
0929 11:22:43.238334 (+ 9287us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:43.238356 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:43.238496 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:43.241000 (+ 2504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:43.241006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:43.241036 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:22:43.241470 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.241482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.244353 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.244359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.246250 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.246263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.248239 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.248247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.248424 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.248458 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:22:43.248548 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:22:43.248710 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9135,"thread_start_us":9894,"threads_started":1}
I20260929 11:22:43.249269 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.229310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.0.97.3:46349)
0929 11:22:43.239492 (+ 10182us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.239498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.239522 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:22:43.239588 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.239593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.239808 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.240501 (+ 693us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:22:43.242646 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.244184 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.246450 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.248042 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.248169 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.248393 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.248653 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.248804 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10031,"thread_start_us":9883,"threads_started":1}
I20260929 11:22:43.253389 12138 tablet_service.cc:1467] Processing CreateTablet for tablet b5786922b2e4452e897ca74024a06ded (DEFAULT_TABLE table=test-table [id=67326c866c8a40fdbda7d2434f5fe840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:22:43.254810 12138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5786922b2e4452e897ca74024a06ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:22:43.257794 12222 tablet_bootstrap.cc:492] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Bootstrap starting.
I20260929 11:22:43.260114 12223 tablet_bootstrap.cc:492] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Bootstrap starting.
I20260929 11:22:43.261811 12222 tablet_bootstrap.cc:654] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Neither blocks nor log segments found. Creating new log.
I20260929 11:22:43.263085 12222 log.cc:824] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:22:43.263475 12223 tablet_bootstrap.cc:654] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:22:43.265018 12223 log.cc:824] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:22:43.268224 12223 tablet_bootstrap.cc:492] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: No bootstrap required, opened a new log
I20260929 11:22:43.268352 12226 tablet_bootstrap.cc:492] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Bootstrap starting.
I20260929 11:22:43.268496 12223 ts_tablet_manager.cc:1397] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 11:22:43.268808 12222 tablet_bootstrap.cc:492] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: No bootstrap required, opened a new log
I20260929 11:22:43.269080 12222 ts_tablet_manager.cc:1397] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.002s
I20260929 11:22:43.271541 12226 tablet_bootstrap.cc:654] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Neither blocks nor log segments found. Creating new log.
I20260929 11:22:43.273007 12226 log.cc:824] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Log is configured to *not* fsync() on all Append() calls
I20260929 11:22:43.276355 12226 tablet_bootstrap.cc:492] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: No bootstrap required, opened a new log
I20260929 11:22:43.276602 12226 ts_tablet_manager.cc:1397] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 11:22:43.276891 12222 raft_consensus.cc:348] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.277216 12222 raft_consensus.cc:374] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:22:43.277338 12222 raft_consensus.cc:729] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3577c41621544ea8a94020eec21dbe9, State: Initialized, Role: FOLLOWER
I20260929 11:22:43.277920 12222 consensus_queue.cc:260] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.279361 12223 raft_consensus.cc:348] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.279722 12223 raft_consensus.cc:374] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:22:43.279811 12223 raft_consensus.cc:729] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dcebe0a40041ae86fe308a376484b9, State: Initialized, Role: FOLLOWER
I20260929 11:22:43.280422 12223 consensus_queue.cc:260] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.281075 12222 ts_tablet_manager.cc:1428] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260929 11:22:43.282482 12069 heartbeater.cc:503] Master 127.0.97.62:42735 was elected leader, sending a full tablet report...
I20260929 11:22:43.283035 12223 ts_tablet_manager.cc:1428] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260929 11:22:43.286957 12226 raft_consensus.cc:348] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.287268 12226 raft_consensus.cc:374] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:22:43.287370 12226 raft_consensus.cc:729] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45892a0c4aee4f12b0c8f0e6e9606860, State: Initialized, Role: FOLLOWER
I20260929 11:22:43.287827 12226 consensus_queue.cc:260] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.288689 12203 heartbeater.cc:503] Master 127.0.97.62:42735 was elected leader, sending a full tablet report...
I20260929 11:22:43.289197 12226 ts_tablet_manager.cc:1428] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260929 11:22:43.324276 12070 tablet.cc:2369] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:22:43.336504 11927 tablet.cc:2369] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:22:43.409514 12204 tablet.cc:2369] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:22:43.514124 12229 raft_consensus.cc:482] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:22:43.514463 12229 raft_consensus.cc:504] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.516438 12229 leader_election.cc:288] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3577c41621544ea8a94020eec21dbe9 (127.0.97.1:43523), 45892a0c4aee4f12b0c8f0e6e9606860 (127.0.97.3:46349)
I20260929 11:22:43.522934 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.516843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36593 (local address 127.0.97.1:43523)
0929 11:22:43.517032 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.517036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.517654 (+ 618us) server_negotiation.cc:407] Connection header received
0929 11:22:43.517894 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.517903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.517979 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.518040 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:22:43.518673 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.519378 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.521225 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.522016 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.522164 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.522277 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.522536 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.522654 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:22:43.523072 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.516842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43523 (local address 127.0.97.2:36593)
0929 11:22:43.517514 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:43.517532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:43.517660 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:43.518218 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:43.518221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:43.518248 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:22:43.518530 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.518538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.519501 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.519509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.521007 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.521021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.522221 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.522227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.522378 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.522404 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:22:43.522515 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 11:22:43.522644 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":1,"thread_start_us":365,"threads_started":1}
I20260929 11:22:43.523124 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.517939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47839 (local address 127.0.97.3:46349)
0929 11:22:43.518130 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.518136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.518271 (+ 135us) server_negotiation.cc:407] Connection header received
0929 11:22:43.518494 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.518500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.518561 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.518654 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:22:43.519151 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.519806 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.521480 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.522239 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.522382 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.522480 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.522743 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.522880 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:22:43.523682 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.517603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46349 (local address 127.0.97.2:47839)
0929 11:22:43.518170 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 11:22:43.518193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:22:43.518360 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:22:43.518669 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:22:43.518672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:22:43.518693 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:22:43.519011 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.519019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.519950 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.519955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.521331 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:22:43.521342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.522440 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.522446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:22:43.522602 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.522625 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:22:43.522722 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:22:43.522844 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260929 11:22:43.524087 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5786922b2e4452e897ca74024a06ded" candidate_uuid: "27dcebe0a40041ae86fe308a376484b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3577c41621544ea8a94020eec21dbe9" is_pre_election: true
I20260929 11:22:43.524720 11881 raft_consensus.cc:2454] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27dcebe0a40041ae86fe308a376484b9 in term 0.
I20260929 11:22:43.524735 12158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5786922b2e4452e897ca74024a06ded" candidate_uuid: "27dcebe0a40041ae86fe308a376484b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" is_pre_election: true
I20260929 11:22:43.525242 12158 raft_consensus.cc:2454] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27dcebe0a40041ae86fe308a376484b9 in term 0.
I20260929 11:22:43.525419 11960 leader_election.cc:302] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dcebe0a40041ae86fe308a376484b9, b3577c41621544ea8a94020eec21dbe9; no voters:
I20260929 11:22:43.525844 12229 raft_consensus.cc:2781] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:22:43.525981 12229 raft_consensus.cc:482] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:22:43.526108 12229 raft_consensus.cc:3037] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:22:43.528671 12229 raft_consensus.cc:504] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.529294 12229 leader_election.cc:288] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [CANDIDATE]: Term 1 election: Requested vote from peers b3577c41621544ea8a94020eec21dbe9 (127.0.97.1:43523), 45892a0c4aee4f12b0c8f0e6e9606860 (127.0.97.3:46349)
I20260929 11:22:43.529883 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5786922b2e4452e897ca74024a06ded" candidate_uuid: "27dcebe0a40041ae86fe308a376484b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3577c41621544ea8a94020eec21dbe9"
I20260929 11:22:43.529883 12158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5786922b2e4452e897ca74024a06ded" candidate_uuid: "27dcebe0a40041ae86fe308a376484b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45892a0c4aee4f12b0c8f0e6e9606860"
I20260929 11:22:43.530097 11881 raft_consensus.cc:3037] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:22:43.530097 12158 raft_consensus.cc:3037] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:22:43.533221 12158 raft_consensus.cc:2454] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27dcebe0a40041ae86fe308a376484b9 in term 1.
I20260929 11:22:43.533212 11881 raft_consensus.cc:2454] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27dcebe0a40041ae86fe308a376484b9 in term 1.
I20260929 11:22:43.533586 11960 leader_election.cc:302] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dcebe0a40041ae86fe308a376484b9, 45892a0c4aee4f12b0c8f0e6e9606860; no voters:
I20260929 11:22:43.533878 12229 raft_consensus.cc:2781] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:22:43.534300 12229 raft_consensus.cc:686] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 1 LEADER]: Becoming Leader. State: Replica: 27dcebe0a40041ae86fe308a376484b9, State: Running, Role: LEADER
I20260929 11:22:43.534830 12229 consensus_queue.cc:237] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } }
I20260929 11:22:43.539403 11712 catalog_manager.cc:5495] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27dcebe0a40041ae86fe308a376484b9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "27dcebe0a40041ae86fe308a376484b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27dcebe0a40041ae86fe308a376484b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260929 11:22:43.599037 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.593444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.0.97.2:40713)
0929 11:22:43.593672 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.593677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.593697 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:22:43.593909 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.593913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.593995 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.594110 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:22:43.594418 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.595107 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.596967 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.597440 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.597543 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.597963 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:22:43.598450 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:22:43.598556 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.598648 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.598711 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:22:43.606858 12158 raft_consensus.cc:1261] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 1 FOLLOWER]: Refusing update from remote peer 27dcebe0a40041ae86fe308a376484b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:22:43.606895 11881 raft_consensus.cc:1261] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 1 FOLLOWER]: Refusing update from remote peer 27dcebe0a40041ae86fe308a376484b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:22:43.607766 12229 consensus_queue.cc:1035] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3577c41621544ea8a94020eec21dbe9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:22:43.608069 12233 consensus_queue.cc:1035] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45892a0c4aee4f12b0c8f0e6e9606860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:22:43.618532 12236 mvcc.cc:205] Tried to move back new op lower bound from 7334629226924498944 to 7334629226642894848. Current Snapshot: MvccSnapshot[applied={T|T < 7334629226924498944}]
I20260929 11:22:43.633095 11712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:22:43.633366 11712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:22:43.636670 11712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b92a7afc4e4b379a5b185422021120: Sending DeleteTablet for 3 replicas of tablet b5786922b2e4452e897ca74024a06ded
I20260929 11:22:43.637725 11861 tablet_service.cc:1514] Processing DeleteTablet for tablet b5786922b2e4452e897ca74024a06ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:22:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39036
I20260929 11:22:43.637807 12004 tablet_service.cc:1514] Processing DeleteTablet for tablet b5786922b2e4452e897ca74024a06ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:22:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50984
I20260929 11:22:43.638070 12138 tablet_service.cc:1514] Processing DeleteTablet for tablet b5786922b2e4452e897ca74024a06ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:22:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55924
I20260929 11:22:43.640398 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 11793
I20260929 11:22:43.652534 12247 tablet_replica.cc:330] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: stopping tablet replica
I20260929 11:22:43.652881 12248 tablet_replica.cc:330] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: stopping tablet replica
I20260929 11:22:43.653501 12246 tablet_replica.cc:330] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: stopping tablet replica
I20260929 11:22:43.653930 12248 raft_consensus.cc:2229] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:22:43.653932 12246 raft_consensus.cc:2229] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:22:43.654430 12246 raft_consensus.cc:2258] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:22:43.654441 12248 raft_consensus.cc:2258] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:22:43.654876 12247 raft_consensus.cc:2229] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:22:43.655524 12247 raft_consensus.cc:2258] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:22:43.656770 12248 ts_tablet_manager.cc:1905] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:22:43.657632 12246 ts_tablet_manager.cc:1905] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:22:43.661068 12247 ts_tablet_manager.cc:1905] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:22:43 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:22:43.665110 12246 ts_tablet_manager.cc:1918] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:22:43.665261 12246 log.cc:1196] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/b5786922b2e4452e897ca74024a06ded
I20260929 11:22:43.665827 12246 ts_tablet_manager.cc:1939] T b5786922b2e4452e897ca74024a06ded P b3577c41621544ea8a94020eec21dbe9: Deleting consensus metadata
I20260929 11:22:43.667716 11700 catalog_manager.cc:4841] TS b3577c41621544ea8a94020eec21dbe9 (127.0.97.1:43523): tablet b5786922b2e4452e897ca74024a06ded (table test-table [id=67326c866c8a40fdbda7d2434f5fe840]) successfully deleted
I20260929 11:22:43.669005 12248 ts_tablet_manager.cc:1918] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:22:43.669184 12248 log.cc:1196] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/b5786922b2e4452e897ca74024a06ded
I20260929 11:22:43.669665 12248 ts_tablet_manager.cc:1939] T b5786922b2e4452e897ca74024a06ded P 45892a0c4aee4f12b0c8f0e6e9606860: Deleting consensus metadata
I20260929 11:22:43.671397 11700 catalog_manager.cc:4841] TS 45892a0c4aee4f12b0c8f0e6e9606860 (127.0.97.3:46349): tablet b5786922b2e4452e897ca74024a06ded (table test-table [id=67326c866c8a40fdbda7d2434f5fe840]) successfully deleted
I20260929 11:22:43.673092 12247 ts_tablet_manager.cc:1918] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:22:43.673254 12247 log.cc:1196] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/b5786922b2e4452e897ca74024a06ded
I20260929 11:22:43.673763 12247 ts_tablet_manager.cc:1939] T b5786922b2e4452e897ca74024a06ded P 27dcebe0a40041ae86fe308a376484b9: Deleting consensus metadata
I20260929 11:22:43.674449 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.642043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.0.97.1:43523)
0929 11:22:43.642245 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.642254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.653506 (+ 11252us) server_negotiation.cc:407] Connection header received
0929 11:22:43.653835 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.653842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.653961 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.654052 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:43.656360 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.657436 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.659559 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.660320 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.660418 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.666663 (+ 6245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:43.666683 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:43.666701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:43.666743 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:43.670713 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.671587 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.671598 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.671601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.671787 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.672218 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.672226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.672229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.672581 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:43.672940 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.673390 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.673625 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:22:43.675158 11921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:22:43.675633 11697 catalog_manager.cc:4841] TS 27dcebe0a40041ae86fe308a376484b9 (127.0.97.2:40713): tablet b5786922b2e4452e897ca74024a06ded (table test-table [id=67326c866c8a40fdbda7d2434f5fe840]) successfully deleted
I20260929 11:22:43.827911 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 11793
I20260929 11:22:43.845968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 11932
Sep 29 11:22:43 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:22:43.863530 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.846835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.97.2:40713)
0929 11:22:43.847023 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.847029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.847117 (+ 88us) server_negotiation.cc:407] Connection header received
0929 11:22:43.847262 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.847268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.847346 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.847452 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:43.848782 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.849996 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.851351 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.852030 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.852162 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:43.857441 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:43.857457 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:43.857473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:43.857512 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:43.860446 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.861302 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.861309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.861311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.861438 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:43.861848 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:43.861851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:43.861853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:43.862205 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:43.862572 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:43.862841 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 11:22:43.863081 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:22:43.864279 12064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:22:43.977017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 11932
I20260929 11:22:43.991165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 12075
Sep 29 11:22:43 dist-test-slave-srst krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:22:44.007579 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:22:43.991910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.0.97.3:46349)
0929 11:22:43.992096 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:22:43.992102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:22:43.992196 (+ 94us) server_negotiation.cc:407] Connection header received
0929 11:22:43.992425 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:22:43.992430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:22:43.992506 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:22:43.992642 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:22:43.993810 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.994536 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.995815 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:22:43.996356 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:22:43.996481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:22:44.002332 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:22:44.002349 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:22:44.002371 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:22:44.002410 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:22:44.005026 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:44.005679 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:44.005690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:44.005692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:44.005852 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:22:44.006201 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:22:44.006204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:22:44.006206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:22:44.006550 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:22:44.006782 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 11:22:44.006980 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 11:22:44.007136 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:22:44.008196 12198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:22:44.095062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 12075
I20260929 11:22:44.111097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 11681
I20260929 11:22:44.112330 11743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:22:44.226831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 11681
I20260929 11:22:44.247586 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 11:22:44.583650 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 11:22:44.583892 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 11:22:44.929265 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 11:22:44.929 UTC [11267] LOG: received smart shutdown request
2026-09-29 11:22:44.933 UTC [11267] LOG: background worker "logical replication launcher" (PID 11276) exited with exit code 1
2026-09-29 11:22:44.933 UTC [11271] LOG: shutting down
2026-09-29 11:22:45.100 UTC [11267] LOG: database system is shut down
2026-09-29T11:22:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47635 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:22:45 dist-test-slave-srst krb5kdc[12262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:22:45 dist-test-slave-srst krb5kdc[12262](info): set up 2 sockets
Sep 29 11:22:45 dist-test-slave-srst krb5kdc[12262](info): commencing operation
krb5kdc: starting...
W20260929 11:22:46.357117 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:22:46 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:22:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 11:22:46.683313 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyNUK1V/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/postgres -l logfile start
2026-09-29 11:22:49.149 UTC [12292] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 11:22:49.150 UTC [12292] LOG: listening on IPv4 address "127.0.97.20", port 33377
2026-09-29 11:22:49.153 UTC [12292] LOG: listening on Unix socket "/tmp/.s.PGSQL.33377"
2026-09-29 11:22:49.175 UTC [12295] LOG: database system was shut down at 2026-09-29 11:22:47 UTC
2026-09-29 11:22:49.179 UTC [12292] LOG: database system is ready to accept connections
I20260929 11:22:50.156574 388 mini_postgres.cc:96] Postgres bound to 33377
2026-09-29 11:22:50.173 UTC [12303] FATAL: database "slave" does not exist
127.0.97.20:33377 - accepting connections
I20260929 11:22:50.174548 388 mini_ranger.cc:161] Starting Ranger...
I20260929 11:22:50.203717 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 11:22:50.669589 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 11:22:50.669708 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-admin
2026-09-29 11:22:50,947 [I] DB FLAVOR :POSTGRES
2026-09-29 11:22:50,947 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:22:50,947 [I] Checking connection..
2026-09-29 11:22:50,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 11:22:51,200 [I] Checking connection passed.
2026-09-29 11:22:51,200 [I] --------- Verifying version history table ---------
2026-09-29 11:22:51,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:22:51,426 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:22:51,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:22:51,653 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:22:51,653 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 11:22:51,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 11:22:51.900 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 11:22:51,906 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 11:22:51,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:22:52,167 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 11:22:52,167 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:22:52,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 11:22:52,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 11:22:52,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srst.c.gcp-upstream.internal','N') ;"
2026-09-29 11:22:52,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:22:53,151 [I] Table x_portal_user does not exist in database ranger
2026-09-29 11:22:53,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:22:53,416 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 11:22:53,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:22:53,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:22:53,896 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 11:22:53,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 11:22:54.805 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.821 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.837 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.852 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.867 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.888 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.953 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.960 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.974 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.984 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:54.994 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:55.006 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:55.016 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:55.022 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:55.028 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:55.034 UTC [12475] WARNING: there is no transaction in progress
2026-09-29 11:22:55,600 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 11:22:55,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 11:22:55,849 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 11:22:55,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:22:56,085 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 11:22:56,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:22:56,324 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 11:22:56,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:22:56,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:22:56,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srst.c.gcp-upstream.internal';"
2026-09-29 11:22:57,062 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 11:22:57,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 11:22:57,311 [I] DB_PATCHES have already been applied
I20260929 11:22:57.317209 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyNUK1V/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 11:22:57.317294 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 11:22:57.331261 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:36163
Sep 29, 2026 11:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 11:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 11:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 11:22:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 11:22:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 11:22:58.434414 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.260s user 0.000s sys 0.040s
Sep 29, 2026 11:23:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 11:23:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 11:23:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 11:23:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 11:23:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 11:23:19.337381 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 11:23:19 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680999, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 11:23:19.498405 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 11:23:19.517741 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 11:23:20.371543 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 11:23:20.371640 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms
I20260929 11:23:20.371695 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33377
2026-09-29 11:23:20,724 [I] DB FLAVOR :POSTGRES
2026-09-29 11:23:20,725 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:23:20,725 [I] Checking connection
2026-09-29 11:23:20,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 11:23:20,951 [I] connection success
2026-09-29 11:23:20,952 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 11:23:20,952 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 11:23:20,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 11:23:21,184 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 11:23:21,184 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:23:21,184 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 11:23:21,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 11:23:21,432 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 11:23:21.855489 388 mini_ranger_kms.cc:310] Created kms service
I20260929 11:23:22.050676 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 11:23:22.096908 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 11:23:22 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790680999, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:22.478130 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 11:23:22.478432 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyNUK1V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyNUK1V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 11:23:22.478493 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 11:23:22.495359 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12655
I20260929 11:23:22.495486 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:34377
Sep 29, 2026 11:23:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:23:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:23:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 11:23:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 11:23:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 11:23:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 11:23:24.765264 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.267s user 0.074s sys 0.293s
I20260929 11:23:24.765508 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 11:23:24.765676 388 mini_ranger_kms.cc:401] 127.0.97.20:34377/kms/v1/keys
Sep 29 11:23:24 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 11:23:25 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 11:23:25 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 11:23:27 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:23:27.808184 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43717
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43717
--ranger_config_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:28.139810 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:28.140087 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:28.140285 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:28.147379 12706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:28.147505 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:28.147585 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:28.147679 12706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:28.147776 12706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:28.157636 12706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:34377/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43717
--ranger_config_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:28.158917 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:28.160202 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:28.171634 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:28.171634 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:28.171845 12714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:28.172376 12706 server_base.cc:1034] running on GCE node
I20260929 11:23:28.173178 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:28.174537 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:28.175757 12706 hybrid_clock.cc:648] HybridClock initialized: now 1790681008175711 us; error 48 us; skew 500 ppm
I20260929 11:23:28.176067 12706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:28 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:28.179118 12706 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:28.180467 12706 webserver.cc:460] Webserver started at http://127.0.97.62:40617/ using document root <none> and password file <none>
I20260929 11:23:28.181080 12706 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:28.181169 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:28.181421 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:23:28 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:28.230273 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "5b03a214fea64f358597cc11966b8108"
format_stamp: "Formatted at 2026-09-29 11:23:28 on dist-test-slave-srst"
server_key: "3b9da77e9d31e16a6f1368a432180359"
server_key_iv: "dd7867675731d0d669abf364d7288439"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:28.231086 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "5b03a214fea64f358597cc11966b8108"
format_stamp: "Formatted at 2026-09-29 11:23:28 on dist-test-slave-srst"
server_key: "3b9da77e9d31e16a6f1368a432180359"
server_key_iv: "dd7867675731d0d669abf364d7288439"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:28.236485 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:23:28.252048 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:28.253656 12706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:23:28.253841 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "5b03a214fea64f358597cc11966b8108"
format_stamp: "Formatted at 2026-09-29 11:23:28 on dist-test-slave-srst"
server_key: "3b9da77e9d31e16a6f1368a432180359"
server_key_iv: "dd7867675731d0d669abf364d7288439"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:28.254079 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:28.294869 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:28.298419 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:28.298745 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:28.316872 12706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43717
I20260929 11:23:28.316891 12772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43717 every 8 connection(s)
I20260929 11:23:28.318120 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:28.320851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 12706
I20260929 11:23:28.321096 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:23:28.321741 388 external_mini_cluster.cc:1428] Setting key 11b78d54b71bcb404539428e18322973
I20260929 11:23:28.321969 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:28.336001 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Bootstrap starting.
Sep 29 11:23:28 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:28.339504 12773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:28.340504 12773 log.cc:824] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:28.343695 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: No bootstrap required, opened a new log
I20260929 11:23:28.344434 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:28.323772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.0.97.62:43717)
0929 11:23:28.324567 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:28.324582 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:28.324660 (+ 78us) server_negotiation.cc:407] Connection header received
0929 11:23:28.326634 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:28.326660 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:28.327123 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:28.327544 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:28.328748 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:28.330825 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:28.332048 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:28.333116 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:28.333325 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:28.337444 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:28.337452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:28.337465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:28.337503 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:28.340396 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:28.341215 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:28.341224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:28.341231 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:28.341441 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:28.341896 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:28.341900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:28.341902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:28.342375 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:28.342601 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:28.343052 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 11:23:28.343325 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":352,"threads_started":1}
I20260929 11:23:28.350991 12773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:28.351368 12773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:28.351473 12773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b03a214fea64f358597cc11966b8108, State: Initialized, Role: FOLLOWER
I20260929 11:23:28.351995 12773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:28.352252 12773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:23:28.352385 12773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:23:28.352528 12773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:28.354562 12773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:28.354959 12773 leader_election.cc:302] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b03a214fea64f358597cc11966b8108; no voters:
I20260929 11:23:28.355437 12773 leader_election.cc:288] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:23:28.355567 12778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:28.356838 12778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 LEADER]: Becoming Leader. State: Replica: 5b03a214fea64f358597cc11966b8108, State: Running, Role: LEADER
I20260929 11:23:28.357368 12778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:28.357740 12773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:28.359757 12780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b03a214fea64f358597cc11966b8108. Latest consensus state: current_term: 1 leader_uuid: "5b03a214fea64f358597cc11966b8108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } } }
I20260929 11:23:28.360098 12780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:28.360610 12779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b03a214fea64f358597cc11966b8108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } } }
I20260929 11:23:28.360944 12779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:28.361151 12784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:28.368618 12784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:28.378683 12773 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:23:28.379626 12784 catalog_manager.cc:1347] Generated new cluster ID: b54acf36f297436bb8f6e2f8fd274300
I20260929 11:23:28.379736 12784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:23:28.422078 12784 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:23:28.424281 12784 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:23:28.434582 12784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Generated new TSK 0
I20260929 11:23:28.435700 12784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:23:29.163582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43717
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:23:29 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 11:23:29.592046 12819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:29.592480 12819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:29.592729 12819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:29.604302 12819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:29.604466 12819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:29.604722 12819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:23:29.620863 12819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:34377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:29.622563 12819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:29.624307 12819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:29.647856 12829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:29.648665 12830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:29.665313 12832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:29.665751 12819 server_base.cc:1034] running on GCE node
I20260929 11:23:29.666623 12819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:29.667871 12819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:29.669106 12819 hybrid_clock.cc:648] HybridClock initialized: now 1790681009669034 us; error 61 us; skew 500 ppm
I20260929 11:23:29.669497 12819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:29 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:29.673477 12819 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:23:29.679656 12819 webserver.cc:460] Webserver started at http://127.0.97.1:43267/ using document root <none> and password file <none>
I20260929 11:23:29.680384 12819 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:29.680507 12819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:29.680843 12819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:23:29 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:29.705240 12819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "8fc1acdbc788413f937e2ad414544793"
format_stamp: "Formatted at 2026-09-29 11:23:29 on dist-test-slave-srst"
server_key: "6df2908c8ab264bcf7a02063b36f5419"
server_key_iv: "1b3117bb98e62dc1cb28f1b438e2a99e"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:29.706326 12819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8fc1acdbc788413f937e2ad414544793"
format_stamp: "Formatted at 2026-09-29 11:23:29 on dist-test-slave-srst"
server_key: "6df2908c8ab264bcf7a02063b36f5419"
server_key_iv: "1b3117bb98e62dc1cb28f1b438e2a99e"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:29.723246 12819 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.003s sys 0.012s
I20260929 11:23:29.752351 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:29.754190 12819 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:23:29.754403 12819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "8fc1acdbc788413f937e2ad414544793"
format_stamp: "Formatted at 2026-09-29 11:23:29 on dist-test-slave-srst"
server_key: "6df2908c8ab264bcf7a02063b36f5419"
server_key_iv: "1b3117bb98e62dc1cb28f1b438e2a99e"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:29.754693 12819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:29.790376 12819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:29.795620 12819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:29.795948 12819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:29.797880 12819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:29.801312 12819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:29.801404 12819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:29.801537 12819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:29.801602 12819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:23:29 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:29.930593 12819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39231
I20260929 11:23:29.930608 12951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39231 every 8 connection(s)
I20260929 11:23:29.931885 12819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:23:29.940929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 12819
I20260929 11:23:29.941120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:23:29.941862 388 external_mini_cluster.cc:1428] Setting key 47d8baa6a0984e96dd8a0a4999457e33
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 29 11:23:29 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:29.991205 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:29.935164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44247 (local address 127.0.97.62:43717)
0929 11:23:29.959960 (+ 24796us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:29.959968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:29.959989 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:23:29.960061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:29.960084 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:29.960183 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:29.960302 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:29.970364 (+ 10062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:29.971427 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:29.973804 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:29.974374 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:29.974514 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:29.984117 (+ 9603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:29.984124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:29.984130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:29.984183 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:29.987542 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:29.988444 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:29.988450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:29.988453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:29.988634 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:29.989171 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:29.989176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:29.989179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:29.989516 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:29.989640 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:29.990603 (+ 963us) server_negotiation.cc:299] Negotiation successful
0929 11:23:29.990825 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24653,"thread_start_us":24542,"threads_started":1}
I20260929 11:23:29.993692 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:29.935740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43717 (local address 127.0.97.1:44247)
0929 11:23:29.954948 (+ 19208us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:29.955030 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:29.957874 (+ 2844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:29.960831 (+ 2957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:29.960844 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:29.968703 (+ 7859us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:29.969687 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:29.969719 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:29.971613 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:29.971619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:29.973655 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:29.973670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:29.974639 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:29.974645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:29.974906 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:29.975675 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:29.975777 (+ 102us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:29.979761 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:29.987711 (+ 7950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:29.987720 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:29.987740 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:29.988299 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:29.988800 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:29.988805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:29.988809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:29.989046 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:29.989728 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:29.989736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:29.990175 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 11:23:29.991744 (+ 1569us) client_negotiation.cc:240] Negotiation successful
0929 11:23:29.992283 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18815,"thread_start_us":22469,"threads_started":1}
I20260929 11:23:29.995731 12952 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43717
I20260929 11:23:30.002369 12952 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:30.003314 12952 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:30.006711 12738 ts_manager.cc:194] Registered new tserver with Master: 8fc1acdbc788413f937e2ad414544793 (127.0.97.1:39231)
I20260929 11:23:30.008939 12738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:23:30.012066 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43717
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:23:30 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 11:23:30.429317 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:30.429750 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:30.429958 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:30.441569 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:30.441709 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:30.441951 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:23:30.457320 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:34377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:30.459476 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:30.461221 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:30.483518 12966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:30.483809 12965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:30.488476 12958 server_base.cc:1034] running on GCE node
W20260929 11:23:30.489226 12968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:30.490111 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:30.491335 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:30.492619 12958 hybrid_clock.cc:648] HybridClock initialized: now 1790681010492546 us; error 41 us; skew 500 ppm
I20260929 11:23:30.492974 12958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:30 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:30.496928 12958 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:23:30.498647 12958 webserver.cc:460] Webserver started at http://127.0.97.2:36003/ using document root <none> and password file <none>
I20260929 11:23:30.499487 12958 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:30.499629 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:30.500015 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:23:30 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:30.533360 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a"
format_stamp: "Formatted at 2026-09-29 11:23:30 on dist-test-slave-srst"
server_key: "c15ed077bda5fb11743d8fcae18a730c"
server_key_iv: "0833f54283cd1de1d607bb3013b867d3"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:30.534495 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a"
format_stamp: "Formatted at 2026-09-29 11:23:30 on dist-test-slave-srst"
server_key: "c15ed077bda5fb11743d8fcae18a730c"
server_key_iv: "0833f54283cd1de1d607bb3013b867d3"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:30.540051 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:23:30.557806 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:30.559423 12958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:23:30.559617 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a"
format_stamp: "Formatted at 2026-09-29 11:23:30 on dist-test-slave-srst"
server_key: "c15ed077bda5fb11743d8fcae18a730c"
server_key_iv: "0833f54283cd1de1d607bb3013b867d3"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:30.559844 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:30.584800 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:30.588819 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:30.589129 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:30.590129 12958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:30.591473 12958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:30.591549 12958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:30.591687 12958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:30.591737 12958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:30.627310 12958 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37119
I20260929 11:23:30.627329 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37119 every 8 connection(s)
I20260929 11:23:30.628628 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:23:30.628911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 12958
I20260929 11:23:30.629087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:23:30.629807 388 external_mini_cluster.cc:1428] Setting key eb74fa5d978fd13b5e17a5e0cba05926
Sep 29 11:23:30 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 11:23:30.671391 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:30.632268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42231 (local address 127.0.97.62:43717)
0929 11:23:30.645056 (+ 12788us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:30.645064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:30.650437 (+ 5373us) server_negotiation.cc:407] Connection header received
0929 11:23:30.653461 (+ 3024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:30.653470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:30.653584 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:30.653706 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:30.655778 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:30.657077 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:30.659789 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:30.660630 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:30.660762 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:30.665393 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:30.665399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:30.665403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:30.665446 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:30.668063 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:30.668821 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:30.668829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:30.668834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:30.668965 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:30.669409 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:30.669413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:30.669416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:30.669813 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:30.669989 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:30.670743 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 11:23:30.671024 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12643,"thread_start_us":12782,"threads_started":1}
I20260929 11:23:30.672346 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:30.632946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43717 (local address 127.0.97.2:42231)
0929 11:23:30.650241 (+ 17295us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:30.650301 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:30.653064 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:30.654006 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:30.654019 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:30.654700 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:30.655569 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:30.655597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:30.657263 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:30.657268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:30.659603 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:30.659618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:30.660856 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:30.660865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:30.661287 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:30.662146 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:30.662187 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:30.665186 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:30.668190 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:30.668200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:30.668218 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:30.668658 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:30.669080 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:30.669085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:30.669089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:30.669290 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:30.669947 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:30.669955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:30.670305 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 11:23:30.670647 (+ 342us) client_negotiation.cc:240] Negotiation successful
0929 11:23:30.671206 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16896,"thread_start_us":16631,"threads_started":1}
I20260929 11:23:30.673846 13095 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43717
I20260929 11:23:30.674175 13095 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:30.674917 13095 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:30.677358 12738 ts_manager.cc:194] Registered new tserver with Master: 42001b3d44ba4d8e9fe4c7f83ba9e25a (127.0.97.2:37119)
I20260929 11:23:30.678179 12738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:23:30.704264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43717
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:31.012997 12952 heartbeater.cc:503] Master 127.0.97.62:43717 was elected leader, sending a full tablet report...
W20260929 11:23:31.053270 13101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:31.053544 13101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:31.053634 13101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:31.061077 13101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:31.061233 13101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:31.061358 13101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:23:31.071135 13101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:34377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:31.072503 13101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:31.073809 13101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:31.087343 13106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:31.087478 13109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:31.087507 13107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:31.088418 13101 server_base.cc:1034] running on GCE node
I20260929 11:23:31.089030 13101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:31.089881 13101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:31.091073 13101 hybrid_clock.cc:648] HybridClock initialized: now 1790681011091017 us; error 45 us; skew 500 ppm
I20260929 11:23:31.091383 13101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:31 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:31.094360 13101 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:23:31.095700 13101 webserver.cc:460] Webserver started at http://127.0.97.3:32789/ using document root <none> and password file <none>
I20260929 11:23:31.096359 13101 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:31.096498 13101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:31.096832 13101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:23:31 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:31.111312 13101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "9de6225dbd644d05a3f86b224afa418e"
format_stamp: "Formatted at 2026-09-29 11:23:31 on dist-test-slave-srst"
server_key: "327200dc23189f71d99b777576be6444"
server_key_iv: "13744b898bcd0cb9fb9183e69262c00d"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:31.112156 13101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9de6225dbd644d05a3f86b224afa418e"
format_stamp: "Formatted at 2026-09-29 11:23:31 on dist-test-slave-srst"
server_key: "327200dc23189f71d99b777576be6444"
server_key_iv: "13744b898bcd0cb9fb9183e69262c00d"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:31.117491 13101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:23:31.130949 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:31.132406 13101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:23:31.132579 13101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "9de6225dbd644d05a3f86b224afa418e"
format_stamp: "Formatted at 2026-09-29 11:23:31 on dist-test-slave-srst"
server_key: "327200dc23189f71d99b777576be6444"
server_key_iv: "13744b898bcd0cb9fb9183e69262c00d"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:31.132849 13101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:31.155479 13101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:31.159117 13101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:31.159416 13101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:31.160296 13101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:31.161653 13101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:31.161746 13101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:31.161881 13101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:31.161957 13101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:31.193243 13101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34693
I20260929 11:23:31.193284 13228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34693 every 8 connection(s)
I20260929 11:23:31.194545 13101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:23:31.198743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13101
I20260929 11:23:31.198916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:23:31.199410 388 external_mini_cluster.cc:1428] Setting key 18582af60932b55bf3b15d5f5c944e6e
Sep 29 11:23:31 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:31.213105 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:31.197033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52359 (local address 127.0.97.62:43717)
0929 11:23:31.197510 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:31.197514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:31.198344 (+ 830us) server_negotiation.cc:407] Connection header received
0929 11:23:31.200276 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:31.200281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:31.200363 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:31.200478 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:31.202044 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.202749 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.204225 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.204713 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.204853 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:31.207962 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:31.207972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:31.207975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:31.208005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:31.210553 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.211166 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.211169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.211171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.211281 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.211620 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.211623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.211624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.211863 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:31.211963 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:31.212538 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 11:23:31.212768 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260929 11:23:31.214455 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:31.197509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43717 (local address 127.0.97.3:52359)
0929 11:23:31.198184 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:31.198227 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:31.199959 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:31.200781 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:31.200794 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:31.201313 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:31.201880 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:31.201897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.202894 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.202899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:31.204082 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:31.204092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.204932 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.204940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:31.205164 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:31.205832 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:31.205857 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:31.207766 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:31.210685 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:31.210690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:31.210703 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:31.211074 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:31.211363 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:31.211366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:31.211368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:31.211537 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:31.211969 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:31.211974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:31.212240 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 11:23:31.212465 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 11:23:31.213040 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
I20260929 11:23:31.215631 13229 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43717
I20260929 11:23:31.215898 13229 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:31.216539 13229 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:31.217870 12738 ts_manager.cc:194] Registered new tserver with Master: 9de6225dbd644d05a3f86b224afa418e (127.0.97.3:34693)
I20260929 11:23:31.218617 12738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52359
I20260929 11:23:31.226706 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:23:31.228288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 12819
Sep 29 11:23:31 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:23:31.243484 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:31.229146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.0.97.1:39231)
0929 11:23:31.229699 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:31.229704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:31.229728 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:23:31.229785 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:31.229791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:31.229961 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:31.230095 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:31.231238 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.232170 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.233579 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.234072 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.234174 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:31.238266 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:31.238276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:31.238286 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:31.238314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:31.240910 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.241549 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.241556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.241560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.241697 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.242058 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.242064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.242067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.242351 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:31.242637 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:31.242904 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:23:31.243132 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":278,"threads_started":1}
I20260929 11:23:31.244573 12947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:31.323998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 12819
I20260929 11:23:31.339669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 12958
Sep 29 11:23:31 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:23:31.354599 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:31.340505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.97.2:37119)
0929 11:23:31.341063 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:31.341070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:31.341094 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:23:31.341162 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:31.341168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:31.341397 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:31.341523 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:31.342612 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.343573 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.345016 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.345528 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.345620 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:31.349775 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:31.349786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:31.349799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:31.349832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:31.352297 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.352954 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.352960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.352962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.353086 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.353383 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.353386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.353391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.353725 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:31.353953 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:31.354147 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 11:23:31.354330 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":328,"threads_started":1}
I20260929 11:23:31.355844 13090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:31.436355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 12958
I20260929 11:23:31.451848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13101
Sep 29 11:23:31 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:23:31.468010 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:31.452815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.0.97.3:34693)
0929 11:23:31.453421 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:31.453427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:31.453474 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:23:31.453539 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:31.453545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:31.453745 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:31.453924 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:31.455267 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.456256 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.457574 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:31.458292 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:31.458421 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:31.462528 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:31.462538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:31.462552 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:31.462581 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:31.465331 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.466013 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.466019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.466021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.466159 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:31.466536 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:31.466544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:31.466546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:31.466850 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:31.467150 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:31.467432 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 11:23:31.467677 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":319,"threads_started":1}
I20260929 11:23:31.469493 13224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:31.552316 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13101
I20260929 11:23:31.568122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 12706
I20260929 11:23:31.569470 12768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:31.673102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 12706
I20260929 11:23:31.693949 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43717
--webserver_interface=127.0.97.62
--webserver_port=40617
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43717
--ranger_config_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:32.011288 13241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:32.011550 13241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:32.011700 13241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:32.018702 13241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:32.018797 13241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:32.018929 13241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:32.019018 13241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:32.019100 13241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:32.029124 13241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43717
--ranger_config_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:32.030438 13241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:32.031735 13241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:32.043709 13246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:32.043672 13249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:32.043620 13247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:32.044474 13241 server_base.cc:1034] running on GCE node
I20260929 11:23:32.045416 13241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:32.046922 13241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:32.048115 13241 hybrid_clock.cc:648] HybridClock initialized: now 1790681012048067 us; error 40 us; skew 500 ppm
I20260929 11:23:32.048420 13241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:32 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:32.051404 13241 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:32.053005 13241 webserver.cc:460] Webserver started at http://127.0.97.62:40617/ using document root <none> and password file <none>
I20260929 11:23:32.053639 13241 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:32.053785 13241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:23:32 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:32.117466 13241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:23:32.121163 13257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:32.122735 13241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 11:23:32.122895 13241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "5b03a214fea64f358597cc11966b8108"
format_stamp: "Formatted at 2026-09-29 11:23:28 on dist-test-slave-srst"
server_key: "3b9da77e9d31e16a6f1368a432180359"
server_key_iv: "dd7867675731d0d669abf364d7288439"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:32.123907 13241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:32.169507 13241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:32.173571 13241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:32.173907 13241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:32.193156 13241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43717
I20260929 11:23:32.193162 13308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43717 every 8 connection(s)
I20260929 11:23:32.194718 13241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:32.196715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13241
I20260929 11:23:32.197697 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39231
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43717
--webserver_port=43267
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:32.204409 13309 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:23:32.208072 13309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Bootstrap starting.
I20260929 11:23:32.226919 13309 log.cc:824] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:32.235658 13309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:23:32.236193 13309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Bootstrap complete.
I20260929 11:23:32.245317 13309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:32.246125 13309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b03a214fea64f358597cc11966b8108, State: Initialized, Role: FOLLOWER
I20260929 11:23:32.246803 13309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:32.247025 13309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:23:32.247130 13309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:23:32.247319 13309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:23:32.251050 13309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:32.251533 13309 leader_election.cc:302] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b03a214fea64f358597cc11966b8108; no voters:
I20260929 11:23:32.252164 13309 leader_election.cc:288] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:23:32.252302 13313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:23:32.253479 13313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [term 2 LEADER]: Becoming Leader. State: Replica: 5b03a214fea64f358597cc11966b8108, State: Running, Role: LEADER
I20260929 11:23:32.254074 13313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } }
I20260929 11:23:32.254428 13309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:32.261971 13315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b03a214fea64f358597cc11966b8108. Latest consensus state: current_term: 2 leader_uuid: "5b03a214fea64f358597cc11966b8108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } } }
I20260929 11:23:32.262431 13315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:32.263012 13314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b03a214fea64f358597cc11966b8108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b03a214fea64f358597cc11966b8108" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43717 } } }
I20260929 11:23:32.263319 13314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:32.269151 13324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:32.274757 13324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:32.276201 13324 catalog_manager.cc:1259] Loaded cluster ID: b54acf36f297436bb8f6e2f8fd274300
I20260929 11:23:32.276360 13324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:23:32.277017 13309 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:23:32.280400 13324 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:23:32.281860 13324 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Loaded TSK: 0
I20260929 11:23:32.283468 13324 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:23:32.648264 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:32.648712 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:32.648873 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:32.660641 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:32.660907 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:32.661141 13311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:23:32.676451 13311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43267
--tserver_master_addrs=127.0.97.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:32.678618 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:32.680323 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:32.712745 13348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:32.713395 13347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:32.713876 13311 server_base.cc:1034] running on GCE node
W20260929 11:23:32.714707 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:32.715490 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:32.722776 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:32.726934 13311 hybrid_clock.cc:648] HybridClock initialized: now 1790681012726792 us; error 70 us; skew 500 ppm
I20260929 11:23:32.727340 13311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:32 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:32.735455 13311 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:23:32.737458 13311 webserver.cc:460] Webserver started at http://127.0.97.1:43267/ using document root <none> and password file <none>
I20260929 11:23:32.738299 13311 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:32.738427 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:23:32 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:32.789899 13311 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:23:32.793030 13358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:32.794574 13311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:23:32.794806 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "8fc1acdbc788413f937e2ad414544793"
format_stamp: "Formatted at 2026-09-29 11:23:29 on dist-test-slave-srst"
server_key: "6df2908c8ab264bcf7a02063b36f5419"
server_key_iv: "1b3117bb98e62dc1cb28f1b438e2a99e"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:32.796530 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:32.832820 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:32.837076 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:32.837375 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:32.838378 13311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:32.841830 13311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:32.841943 13311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:32.842053 13311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:32.842136 13311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:32.893126 13311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39231
I20260929 11:23:32.893177 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39231 every 8 connection(s)
I20260929 11:23:32.894625 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:23:32.895915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13311
I20260929 11:23:32.898038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37119
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43717
--webserver_port=36003
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:23:32 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:32.928481 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:32.898586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43717 (local address 127.0.97.1:47345)
0929 11:23:32.900030 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:32.900075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:32.903122 (+ 3047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:32.907065 (+ 3943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:32.907076 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:32.907718 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:32.908502 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:32.908528 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:32.911279 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:32.911284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:32.912538 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:32.912550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:32.914425 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:32.914434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:32.914763 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:32.915544 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:32.915577 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:32.918550 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:32.923235 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:32.923244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:32.923261 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:32.923903 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:32.924458 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:32.924463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:32.924466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:32.924742 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:32.925837 (+ 1095us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:32.925845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:32.926223 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 11:23:32.926529 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 11:23:32.927096 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":1574,"threads_started":1}
I20260929 11:23:32.931074 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:32.899951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47345 (local address 127.0.97.62:43717)
0929 11:23:32.904680 (+ 4729us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:32.904703 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:32.904757 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:23:32.905405 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:32.905428 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:32.905889 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:32.906333 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:32.908956 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:32.911009 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:32.912774 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:32.913864 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:32.914061 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:32.918851 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:32.918864 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:32.918886 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:32.918951 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:32.923070 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:32.924086 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:32.924095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:32.924104 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:32.924313 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:32.924906 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:32.924913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:32.924919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:32.925664 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:32.925897 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:32.929393 (+ 3496us) server_negotiation.cc:299] Negotiation successful
0929 11:23:32.929839 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4453,"thread_start_us":4271,"threads_started":1}
I20260929 11:23:32.934427 13472 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43717
I20260929 11:23:32.934746 13472 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:32.935624 13472 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
W20260929 11:23:32.938759 13472 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43717 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:23:32.938984 13472 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:32.939240 13472 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:32.940375 13472 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:32.940663 13472 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
W20260929 11:23:32.941737 13472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:23:33.292750 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:33.293030 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:33.293185 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:33.301067 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:33.301194 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:33.301360 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:23:33.311477 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36003
--tserver_master_addrs=127.0.97.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:33.312796 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:33.314136 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:33.326700 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:33.327463 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:33.327257 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:33.335508 13476 server_base.cc:1034] running on GCE node
I20260929 11:23:33.336397 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:33.347258 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:33.350705 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790681013350595 us; error 99 us; skew 500 ppm
I20260929 11:23:33.351133 13476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:33 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:33.355346 13476 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:23:33.365235 13476 webserver.cc:460] Webserver started at http://127.0.97.2:36003/ using document root <none> and password file <none>
I20260929 11:23:33.366183 13476 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:33.366334 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:23:33 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:33.399274 13476 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260929 11:23:33.403123 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:33.405084 13476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:23:33.405314 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a"
format_stamp: "Formatted at 2026-09-29 11:23:30 on dist-test-slave-srst"
server_key: "c15ed077bda5fb11743d8fcae18a730c"
server_key_iv: "0833f54283cd1de1d607bb3013b867d3"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:33.406656 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:33.427332 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:33.432623 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:33.432924 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:33.434057 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:33.435729 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:33.435858 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:33.436008 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:33.436100 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 11:23:33 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:33.519713 13476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37119
I20260929 11:23:33.519865 13616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37119 every 8 connection(s)
I20260929 11:23:33.521420 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:23:33.530506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13476
I20260929 11:23:33.531224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34693
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43717
--webserver_port=32789
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:23:33 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:33.559808 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:33.528377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38835 (local address 127.0.97.62:43717)
0929 11:23:33.539838 (+ 11461us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:33.539857 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:33.540669 (+ 812us) server_negotiation.cc:407] Connection header received
0929 11:23:33.543304 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:33.543311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:33.543429 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:33.543582 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:33.545396 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:33.546197 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:33.547881 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:33.548302 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:33.548437 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:33.552543 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:33.552550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:33.552554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:33.552616 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:33.556229 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:33.557050 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:33.557077 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:33.557082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:33.557248 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:33.557779 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:33.557786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:33.557789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:33.558132 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:33.558282 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:33.559227 (+ 945us) server_negotiation.cc:299] Negotiation successful
0929 11:23:33.559442 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11318,"thread_start_us":11478,"threads_started":1}
I20260929 11:23:33.561512 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:33.527783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43717 (local address 127.0.97.2:38835)
0929 11:23:33.540454 (+ 12671us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:33.540504 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:33.542937 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:33.543878 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:33.543893 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:33.544538 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:33.545200 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:33.545217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:33.546346 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:33.546352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:33.547732 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:33.547743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:33.548531 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:33.548539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:33.548808 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:33.549542 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:33.549576 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:33.552287 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:33.556361 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:33.556369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:33.556386 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:33.556925 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:33.557377 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:33.557381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:33.557384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:33.557644 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:33.558352 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:33.558361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:33.558790 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 11:23:33.559138 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 11:23:33.559647 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12352,"thread_start_us":13085,"threads_started":1}
I20260929 11:23:33.563639 13617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43717
I20260929 11:23:33.564000 13617 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:33.565021 13617 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:33.567037 13274 ts_manager.cc:194] Registered new tserver with Master: 42001b3d44ba4d8e9fe4c7f83ba9e25a (127.0.97.2:37119)
I20260929 11:23:33.569217 13274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38835
Sep 29 11:23:33 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:33.941880 13472 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:33.942338 13472 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:33.943689 13274 ts_manager.cc:194] Registered new tserver with Master: 8fc1acdbc788413f937e2ad414544793 (127.0.97.1:39231)
I20260929 11:23:33.945071 13274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47345
W20260929 11:23:34.009963 13621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:34.010346 13621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:34.010491 13621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:34.021485 13621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:34.021616 13621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:34.021785 13621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:23:34.037405 13621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:34377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32789
--tserver_master_addrs=127.0.97.62:43717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:34.039089 13621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:34.040736 13621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:34.057106 13626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:34.057083 13627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:34.062722 13621 server_base.cc:1034] running on GCE node
W20260929 11:23:34.063045 13629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:34.063810 13621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:34.065109 13621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:34.066351 13621 hybrid_clock.cc:648] HybridClock initialized: now 1790681014066281 us; error 59 us; skew 500 ppm
I20260929 11:23:34.066740 13621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:34 dist-test-slave-srst krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:34.074651 13621 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:23:34.076674 13621 webserver.cc:460] Webserver started at http://127.0.97.3:32789/ using document root <none> and password file <none>
I20260929 11:23:34.077620 13621 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:34.077775 13621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:23:34 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:34.128706 13621 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.006s sys 0.009s
I20260929 11:23:34.133019 13637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:34.134541 13621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:23:34.134718 13621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "9de6225dbd644d05a3f86b224afa418e"
format_stamp: "Formatted at 2026-09-29 11:23:31 on dist-test-slave-srst"
server_key: "327200dc23189f71d99b777576be6444"
server_key_iv: "13744b898bcd0cb9fb9183e69262c00d"
server_key_version: "kuduclusterkey@0"
I20260929 11:23:34.136230 13621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:34.172644 13621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 29 11:23:34 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 11:23:34.179929 13621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:34.180296 13621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:34.197407 13621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:34.199311 13621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:34.199440 13621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:34.199559 13621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:34.199646 13621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:34.531396 13621 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34693
I20260929 11:23:34.531454 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34693 every 8 connection(s)
I20260929 11:23:34.533290 13621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:23:34.535728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13621
Sep 29 11:23:34 dist-test-slave-srst krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:34.556209 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.536753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.0.97.62:43717)
0929 11:23:34.539802 (+ 3049us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.539806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.540620 (+ 814us) server_negotiation.cc:407] Connection header received
0929 11:23:34.540838 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.540844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.540925 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.541015 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:34.543205 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.544131 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.545970 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.546695 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.546773 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.550099 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:34.550104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:34.550107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:34.550141 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:34.553174 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.553822 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.553832 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.553835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.553977 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.554424 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.554435 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.554438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.554767 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:34.555113 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.555461 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.555731 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2936,"thread_start_us":289,"threads_started":1}
I20260929 11:23:34.559363 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.536031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48209 (local address 127.0.97.62:43717)
0929 11:23:34.537264 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.537268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.540659 (+ 3391us) server_negotiation.cc:407] Connection header received
0929 11:23:34.543408 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.543412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.543493 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.543597 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:34.545369 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.546441 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.548830 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.549259 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.549347 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.553122 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:34.553127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:34.553130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:34.553158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:34.556068 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.556916 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.556919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.556921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.557018 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.557549 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.557554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.557557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.557872 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:34.558024 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.558835 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.559033 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":239,"threads_started":1}
I20260929 11:23:34.560345 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.536593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43717 (local address 127.0.97.3:48209)
0929 11:23:34.540396 (+ 3803us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:34.540455 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:34.543139 (+ 2684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:34.543795 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:34.543804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:34.544331 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:34.545181 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.545218 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.546612 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.546616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.548684 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.548702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.549498 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.549507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.549822 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.550619 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:34.550643 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:34.552827 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:34.556200 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:34.556211 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:34.556229 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:34.556782 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:34.557128 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:34.557133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:34.557137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:34.557416 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:34.558036 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:34.558045 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:34.558462 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 11:23:34.558772 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 11:23:34.559323 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3453,"thread_start_us":3221,"threads_started":1}
I20260929 11:23:34.561754 13758 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43717
I20260929 11:23:34.562112 13758 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:34.562830 13758 heartbeater.cc:511] Master 127.0.97.62:43717 requested a full tablet report, sending...
I20260929 11:23:34.564237 13274 ts_manager.cc:194] Registered new tserver with Master: 9de6225dbd644d05a3f86b224afa418e (127.0.97.3:34693)
I20260929 11:23:34.565147 13274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48209
I20260929 11:23:34.571384 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:23:34.573865 13617 heartbeater.cc:503] Master 127.0.97.62:43717 was elected leader, sending a full tablet report...
I20260929 11:23:34.586565 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.574336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.0.97.62:43717)
0929 11:23:34.574504 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.574509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.575143 (+ 634us) server_negotiation.cc:407] Connection header received
0929 11:23:34.575300 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.575308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.575404 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.575530 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:34.576942 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.577644 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.578918 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.579301 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.579386 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.581837 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:34.581841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:34.581845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:34.581872 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:34.584240 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.584895 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.584900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.584902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.585038 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.585381 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.585385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.585387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.585639 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:34.585818 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.586049 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.586208 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:23:34.591291 13274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:23:34.598483 13274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:23:34.624780 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.611706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43884)
0929 11:23:34.614906 (+ 3200us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:34.614921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:34.615024 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:34.616609 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:34.616617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:34.616684 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:34.618079 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.618094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.619841 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.619847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.621920 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.621934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.623770 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.623777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.623960 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.624016 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 11:23:34.624152 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:23:34.624332 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3082,"mutex_wait_us":143,"spinlock_wait_cycles":9344,"thread_start_us":2936,"threads_started":1}
I20260929 11:23:34.624794 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.611783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.97.1:39231)
0929 11:23:34.615621 (+ 3838us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.615630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.615655 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:23:34.615752 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.615762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.616055 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.616266 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:34.618308 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.619705 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.622147 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.623533 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.623705 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.623929 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.624236 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.624417 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3643,"thread_start_us":3495,"threads_started":1}
I20260929 11:23:34.624951 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.612257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.0.97.3:34693)
0929 11:23:34.613610 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.613623 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.614460 (+ 837us) server_negotiation.cc:407] Connection header received
0929 11:23:34.615666 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.615679 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.615968 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.616191 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:34.618243 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.619747 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.622288 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.623586 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.623717 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.623927 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.624455 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.624640 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":1010,"threads_started":1}
I20260929 11:23:34.625638 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.611693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34693 (local address 127.0.0.1:44638)
0929 11:23:34.614207 (+ 2514us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:34.614240 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:34.614402 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:34.616583 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:34.616615 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:34.616679 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:34.618021 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.618038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.619884 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.619893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.621847 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.621857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.624075 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.624079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.624259 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.624295 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:23:34.625135 (+ 840us) client_negotiation.cc:240] Negotiation successful
0929 11:23:34.625276 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2249,"mutex_wait_us":93,"spinlock_wait_cycles":17536,"thread_start_us":2129,"threads_started":1}
I20260929 11:23:34.625895 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.615438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.0.97.2:37119)
0929 11:23:34.617339 (+ 1901us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.617351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.618234 (+ 883us) server_negotiation.cc:407] Connection header received
0929 11:23:34.618326 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.618334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.618545 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.618716 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:34.619462 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.621013 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.623233 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.624474 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.624611 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.624821 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.625391 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.625575 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":1519,"threads_started":1}
I20260929 11:23:34.626358 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37119 (local address 127.0.0.1:60746)
0929 11:23:34.617941 (+ 2262us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:34.617980 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:34.618109 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:34.618931 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:34.618936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:34.618957 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:34.619284 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.619295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.621150 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.621155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.623101 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.623112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.624988 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.624995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.625145 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.625177 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:23:34.625268 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:23:34.626012 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"mutex_wait_us":45,"thread_start_us":1951,"threads_started":1}
I20260929 11:23:34.629172 13686 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcc9292268b40958ee9830b474f612a (DEFAULT_TABLE table=test-table [id=748fa13290b544fa85c861225f2e7704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:34.629184 13407 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcc9292268b40958ee9830b474f612a (DEFAULT_TABLE table=test-table [id=748fa13290b544fa85c861225f2e7704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:34.629647 13551 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcc9292268b40958ee9830b474f612a (DEFAULT_TABLE table=test-table [id=748fa13290b544fa85c861225f2e7704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:34.630362 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcc9292268b40958ee9830b474f612a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:34.630532 13407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcc9292268b40958ee9830b474f612a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:34.630801 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcc9292268b40958ee9830b474f612a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:34.641607 13779 tablet_bootstrap.cc:492] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Bootstrap starting.
I20260929 11:23:34.644785 13780 tablet_bootstrap.cc:492] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: Bootstrap starting.
I20260929 11:23:34.645192 13779 tablet_bootstrap.cc:654] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:34.646448 13779 log.cc:824] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:34.647732 13781 tablet_bootstrap.cc:492] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Bootstrap starting.
I20260929 11:23:34.648186 13780 tablet_bootstrap.cc:654] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:34.649308 13779 tablet_bootstrap.cc:492] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: No bootstrap required, opened a new log
I20260929 11:23:34.649466 13780 log.cc:824] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:34.649585 13779 ts_tablet_manager.cc:1397] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 11:23:34.652379 13780 tablet_bootstrap.cc:492] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: No bootstrap required, opened a new log
I20260929 11:23:34.652518 13781 tablet_bootstrap.cc:654] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:34.652681 13780 ts_tablet_manager.cc:1397] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260929 11:23:34.653692 13781 log.cc:824] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:34.656911 13781 tablet_bootstrap.cc:492] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: No bootstrap required, opened a new log
I20260929 11:23:34.657157 13781 ts_tablet_manager.cc:1397] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 11:23:34.657477 13779 raft_consensus.cc:348] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.657826 13779 raft_consensus.cc:374] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:34.657955 13779 raft_consensus.cc:729] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de6225dbd644d05a3f86b224afa418e, State: Initialized, Role: FOLLOWER
I20260929 11:23:34.658596 13779 consensus_queue.cc:260] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.659641 13758 heartbeater.cc:503] Master 127.0.97.62:43717 was elected leader, sending a full tablet report...
I20260929 11:23:34.660279 13779 ts_tablet_manager.cc:1428] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 11:23:34.662942 13780 raft_consensus.cc:348] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.663336 13780 raft_consensus.cc:374] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:34.663431 13780 raft_consensus.cc:729] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc1acdbc788413f937e2ad414544793, State: Initialized, Role: FOLLOWER
I20260929 11:23:34.664039 13780 consensus_queue.cc:260] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.665138 13472 heartbeater.cc:503] Master 127.0.97.62:43717 was elected leader, sending a full tablet report...
I20260929 11:23:34.666930 13780 ts_tablet_manager.cc:1428] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 11:23:34.668488 13781 raft_consensus.cc:348] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.668897 13781 raft_consensus.cc:374] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:34.668989 13781 raft_consensus.cc:729] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42001b3d44ba4d8e9fe4c7f83ba9e25a, State: Initialized, Role: FOLLOWER
I20260929 11:23:34.669608 13781 consensus_queue.cc:260] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.671224 13781 ts_tablet_manager.cc:1428] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260929 11:23:34.785010 13759 tablet.cc:2369] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:23:34.788265 13618 tablet.cc:2369] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:23:34.822551 13787 raft_consensus.cc:482] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:23:34.822912 13787 raft_consensus.cc:504] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.824890 13787 leader_election.cc:288] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fc1acdbc788413f937e2ad414544793 (127.0.97.1:39231), 9de6225dbd644d05a3f86b224afa418e (127.0.97.3:34693)
I20260929 11:23:34.831862 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.825587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45973 (local address 127.0.97.3:34693)
0929 11:23:34.825826 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.825833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.826275 (+ 442us) server_negotiation.cc:407] Connection header received
0929 11:23:34.826478 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.826484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.826586 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.826694 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:34.827295 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.828333 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.830261 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.831000 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.831140 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.831243 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.831502 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.831627 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 11:23:34.831923 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.825193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34693 (local address 127.0.97.2:45973)
0929 11:23:34.826126 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:34.826149 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:34.826327 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:34.826725 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:34.826730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:34.826755 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:34.827141 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.827151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.828467 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.828475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.830107 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.830125 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.831200 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.831206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.831349 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.831375 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:23:34.831480 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:23:34.831596 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":560,"threads_started":1}
I20260929 11:23:34.832165 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.825310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33569 (local address 127.0.97.1:39231)
0929 11:23:34.825558 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.825567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.826465 (+ 898us) server_negotiation.cc:407] Connection header received
0929 11:23:34.826737 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.826742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.826820 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.826940 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:34.827610 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.828363 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.830401 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.831209 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.831369 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.831484 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.831782 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.831910 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 11:23:34.832475 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.825298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.97.2:33569)
0929 11:23:34.826328 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:34.826352 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:34.826540 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:34.826993 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:34.826996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:34.827021 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:34.827451 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.827466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.828543 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.828549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.830239 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:34.830251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.831436 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.831443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:34.831606 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.831630 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:23:34.831756 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 11:23:34.831874 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":694,"threads_started":1}
I20260929 11:23:34.833178 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcc9292268b40958ee9830b474f612a" candidate_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de6225dbd644d05a3f86b224afa418e" is_pre_election: true
I20260929 11:23:34.833441 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcc9292268b40958ee9830b474f612a" candidate_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc1acdbc788413f937e2ad414544793" is_pre_election: true
I20260929 11:23:34.833712 13708 raft_consensus.cc:2454] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42001b3d44ba4d8e9fe4c7f83ba9e25a in term 0.
I20260929 11:23:34.833906 13427 raft_consensus.cc:2454] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42001b3d44ba4d8e9fe4c7f83ba9e25a in term 0.
I20260929 11:23:34.834524 13507 leader_election.cc:302] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42001b3d44ba4d8e9fe4c7f83ba9e25a, 9de6225dbd644d05a3f86b224afa418e; no voters:
I20260929 11:23:34.834950 13787 raft_consensus.cc:2781] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:23:34.835106 13787 raft_consensus.cc:482] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:23:34.835222 13787 raft_consensus.cc:3037] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:34.837491 13787 raft_consensus.cc:504] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.838086 13787 leader_election.cc:288] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [CANDIDATE]: Term 1 election: Requested vote from peers 8fc1acdbc788413f937e2ad414544793 (127.0.97.1:39231), 9de6225dbd644d05a3f86b224afa418e (127.0.97.3:34693)
I20260929 11:23:34.838459 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcc9292268b40958ee9830b474f612a" candidate_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc1acdbc788413f937e2ad414544793"
I20260929 11:23:34.838541 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcc9292268b40958ee9830b474f612a" candidate_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de6225dbd644d05a3f86b224afa418e"
I20260929 11:23:34.838675 13427 raft_consensus.cc:3037] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:34.838733 13708 raft_consensus.cc:3037] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:34.842170 13427 raft_consensus.cc:2454] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42001b3d44ba4d8e9fe4c7f83ba9e25a in term 1.
I20260929 11:23:34.842183 13708 raft_consensus.cc:2454] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42001b3d44ba4d8e9fe4c7f83ba9e25a in term 1.
I20260929 11:23:34.842590 13505 leader_election.cc:302] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42001b3d44ba4d8e9fe4c7f83ba9e25a, 8fc1acdbc788413f937e2ad414544793; no voters:
I20260929 11:23:34.842896 13787 raft_consensus.cc:2781] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:34.843734 13787 raft_consensus.cc:686] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 1 LEADER]: Becoming Leader. State: Replica: 42001b3d44ba4d8e9fe4c7f83ba9e25a, State: Running, Role: LEADER
I20260929 11:23:34.844188 13787 consensus_queue.cc:237] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } }
I20260929 11:23:34.848946 13273 catalog_manager.cc:5495] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 42001b3d44ba4d8e9fe4c7f83ba9e25a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42001b3d44ba4d8e9fe4c7f83ba9e25a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37119 } health_report { overall_health: HEALTHY } } }
W20260929 11:23:34.906651 13473 tablet.cc:2369] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:23:34.915448 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.910143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.0.97.2:37119)
0929 11:23:34.910361 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.910369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.910388 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:23:34.910676 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.910688 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.910769 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.910917 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:23:34.911307 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.912006 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.913434 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.913800 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.913923 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.914418 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:23:34.914894 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:23:34.915061 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.915111 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.915184 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:23:34.922945 13708 raft_consensus.cc:1261] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 1 FOLLOWER]: Refusing update from remote peer 42001b3d44ba4d8e9fe4c7f83ba9e25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:34.922945 13427 raft_consensus.cc:1261] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793 [term 1 FOLLOWER]: Refusing update from remote peer 42001b3d44ba4d8e9fe4c7f83ba9e25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:34.923852 13787 consensus_queue.cc:1035] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc1acdbc788413f937e2ad414544793" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:34.924142 13790 consensus_queue.cc:1035] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de6225dbd644d05a3f86b224afa418e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:34.938477 13793 mvcc.cc:205] Tried to move back new op lower bound from 7334629437115953152 to 7334629436805668864. Current Snapshot: MvccSnapshot[applied={T|T < 7334629437115953152}]
I20260929 11:23:34.945358 13795 mvcc.cc:205] Tried to move back new op lower bound from 7334629437115953152 to 7334629436805668864. Current Snapshot: MvccSnapshot[applied={T|T < 7334629437115953152}]
I20260929 11:23:34.955755 13274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:23:34.956040 13274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:23:34.959442 13274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b03a214fea64f358597cc11966b8108: Sending DeleteTablet for 3 replicas of tablet 7bcc9292268b40958ee9830b474f612a
I20260929 11:23:34.960493 13686 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bcc9292268b40958ee9830b474f612a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44638
I20260929 11:23:34.960566 13407 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bcc9292268b40958ee9830b474f612a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43884
I20260929 11:23:34.961239 13551 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bcc9292268b40958ee9830b474f612a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60746
I20260929 11:23:34.962436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13311
I20260929 11:23:34.965329 13805 tablet_replica.cc:330] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: stopping tablet replica
I20260929 11:23:34.965754 13805 raft_consensus.cc:2229] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:23:34.966423 13805 raft_consensus.cc:2258] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:34.972638 13805 ts_tablet_manager.cc:1905] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:34.982369 13805 ts_tablet_manager.cc:1918] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:34.982692 13805 log.cc:1196] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/7bcc9292268b40958ee9830b474f612a
I20260929 11:23:34.983461 13805 ts_tablet_manager.cc:1939] T 7bcc9292268b40958ee9830b474f612a P 42001b3d44ba4d8e9fe4c7f83ba9e25a: Deleting consensus metadata
I20260929 11:23:34.984210 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:34.964375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.97.1:39231)
0929 11:23:34.964534 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:34.964540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:34.964557 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:23:34.964641 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:34.964647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:34.964728 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:34.964830 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:34.966640 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.967761 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.969376 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:34.970100 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:34.970233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:34.975845 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:34.975865 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:34.975890 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:34.975942 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:34.980918 (+ 4976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.981735 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.981748 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.981754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.981957 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:34.982371 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:34.982378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:34.982384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:34.982968 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:34.983264 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:34.983526 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 11:23:34.983781 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:23:34.985020 13467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:34.986238 13259 catalog_manager.cc:4841] TS 42001b3d44ba4d8e9fe4c7f83ba9e25a (127.0.97.2:37119): tablet 7bcc9292268b40958ee9830b474f612a (table test-table [id=748fa13290b544fa85c861225f2e7704]) successfully deleted
I20260929 11:23:34.987346 13803 tablet_replica.cc:330] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: stopping tablet replica
I20260929 11:23:34.988581 13803 raft_consensus.cc:2229] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:23:34.989733 13803 raft_consensus.cc:2258] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:34.991746 13803 ts_tablet_manager.cc:1905] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:34.998811 13803 ts_tablet_manager.cc:1918] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:34.998965 13803 log.cc:1196] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/7bcc9292268b40958ee9830b474f612a
I20260929 11:23:34.999478 13803 ts_tablet_manager.cc:1939] T 7bcc9292268b40958ee9830b474f612a P 9de6225dbd644d05a3f86b224afa418e: Deleting consensus metadata
I20260929 11:23:35.001487 13261 catalog_manager.cc:4841] TS 9de6225dbd644d05a3f86b224afa418e (127.0.97.3:34693): tablet 7bcc9292268b40958ee9830b474f612a (table test-table [id=748fa13290b544fa85c861225f2e7704]) successfully deleted
I20260929 11:23:35.081522 13804 tablet_replica.cc:330] T 7bcc9292268b40958ee9830b474f612a P 8fc1acdbc788413f937e2ad414544793: stopping tablet replica
I20260929 11:23:35.081755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13311
W20260929 11:23:35.104463 13259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39231 (error 108)
I20260929 11:23:35.105285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13476
I20260929 11:23:35.106437 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.105865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43914)
0929 11:23:35.106131 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.106275 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260929 11:23:35.106714 13259 catalog_manager.cc:4584] TS 8fc1acdbc788413f937e2ad414544793 (127.0.97.1:39231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bcc9292268b40958ee9830b474f612a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
I20260929 11:23:35.129268 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.106029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.0.97.2:37119)
0929 11:23:35.106284 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:35.106292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:35.106337 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:23:35.106556 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:35.106560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:35.106644 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:35.106761 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:35.108388 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:35.109383 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:35.111212 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:35.111763 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:35.111855 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:35.123079 (+ 11224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:35.123094 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:35.123111 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:35.123146 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:35.126383 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:35.127103 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:35.127110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:35.127112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:35.127244 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:35.127744 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:35.127747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:35.127749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:35.128133 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:35.128370 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:35.128753 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 11:23:35.128929 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 11:23:35.130018 13612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:35.160416 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.159893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43916)
0929 11:23:35.160116 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.160225 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 11:23:35.161526 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.161155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43918)
0929 11:23:35.161309 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.161402 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:23:35.222019 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.221463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43930)
0929 11:23:35.221712 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.221818 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 11:23:35.223160 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.222792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43940)
0929 11:23:35.222952 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.223045 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:23:35.238998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13476
I20260929 11:23:35.257018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13621
I20260929 11:23:35.269562 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.257752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.0.97.3:34693)
0929 11:23:35.257920 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:35.257927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:35.257946 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:23:35.258150 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:35.258154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:35.258219 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:35.258316 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:35.259506 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:35.260248 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:35.261500 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:35.261883 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:35.262006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:35.264275 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:35.264288 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:35.264304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:35.264334 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:35.266853 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:35.267513 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:35.267520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:35.267522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:35.267655 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:35.268007 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:35.268012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:35.268015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:35.268358 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:35.268691 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:35.268988 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 11:23:35.269208 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:23:35.270203 13753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:35.304155 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.303564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43942)
0929 11:23:35.303805 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.303916 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 11:23:35.305359 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.304931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43958)
0929 11:23:35.305140 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.305238 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 11:23:35.355508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13621
I20260929 11:23:35.371816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13241
I20260929 11:23:35.373010 13304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:35.482199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13241
I20260929 11:23:35.482249 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:35.481508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39231 (local address 127.0.0.1:43972)
0929 11:23:35.481869 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:35.482024 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260929 11:23:35.502796 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 11:23:35.837587 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 11:23:35.837837 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 11:23:36.184000 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 11:23:36.184 UTC [12292] LOG: received smart shutdown request
2026-09-29 11:23:36.187 UTC [12292] LOG: background worker "logical replication launcher" (PID 12301) exited with exit code 1
2026-09-29 11:23:36.187 UTC [12296] LOG: shutting down
2026-09-29 11:23:36.347 UTC [12292] LOG: database system is shut down
2026-09-29T11:23:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51239 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:23:36 dist-test-slave-srst krb5kdc[13819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:23:36 dist-test-slave-srst krb5kdc[13819](info): set up 2 sockets
Sep 29 11:23:36 dist-test-slave-srst krb5kdc[13819](info): commencing operation
krb5kdc: starting...
W20260929 11:23:37.557278 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:23:37 dist-test-slave-srst krb5kdc[13819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:23:37.776515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45051
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45051,127.0.97.61:36291,127.0.97.60:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:38.097920 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:38.098201 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:38.098307 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:38.105574 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:38.105690 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:38.105737 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:38.105790 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:38.105836 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:38.115736 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45051,127.0.97.61:36291,127.0.97.60:39827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:38.117079 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:38.118317 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:38.129895 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:38.129962 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:38.130088 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:38.130867 13832 server_base.cc:1034] running on GCE node
I20260929 11:23:38.131655 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:38.133034 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:38.134229 13832 hybrid_clock.cc:648] HybridClock initialized: now 1790681018134186 us; error 31 us; skew 500 ppm
I20260929 11:23:38.134593 13832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:38 dist-test-slave-srst krb5kdc[13819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:38.137779 13832 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:38.139204 13832 webserver.cc:460] Webserver started at http://127.0.97.62:43809/ using document root <none> and password file <none>
I20260929 11:23:38.139824 13832 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:38.139919 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:38.140172 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:38.143097 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "0699265d4c4f4703be44e9b89aed6df1"
format_stamp: "Formatted at 2026-09-29 11:23:38 on dist-test-slave-srst"
server_key: "a37d82baf792565f845c997ce24f7807"
server_key_iv: "de6879d6bd3ededb778b6f97bff156f8"
server_key_version: "encryptionkey@0"
I20260929 11:23:38.143808 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "0699265d4c4f4703be44e9b89aed6df1"
format_stamp: "Formatted at 2026-09-29 11:23:38 on dist-test-slave-srst"
server_key: "a37d82baf792565f845c997ce24f7807"
server_key_iv: "de6879d6bd3ededb778b6f97bff156f8"
server_key_version: "encryptionkey@0"
I20260929 11:23:38.148831 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:23:38.152083 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:38.153437 13832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:23:38.153640 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "0699265d4c4f4703be44e9b89aed6df1"
format_stamp: "Formatted at 2026-09-29 11:23:38 on dist-test-slave-srst"
server_key: "a37d82baf792565f845c997ce24f7807"
server_key_iv: "de6879d6bd3ededb778b6f97bff156f8"
server_key_version: "encryptionkey@0"
I20260929 11:23:38.153841 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:38.167531 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:38.171391 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:38.171672 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:38.190373 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45051
I20260929 11:23:38.190405 13898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45051 every 8 connection(s)
I20260929 11:23:38.191761 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:38.195570 13899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:38.196431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13832
I20260929 11:23:38.196686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:23:38.197314 388 external_mini_cluster.cc:1428] Setting key 8957a890ddb87c75ae76b356c865522d
I20260929 11:23:38.202052 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 11:23:38 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:38.229983 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.205669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.0.97.62:45051)
0929 11:23:38.207545 (+ 1876us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:38.207550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:38.207591 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:23:38.209903 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:38.209909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:38.210137 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:38.210286 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:38.211937 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.213179 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.215651 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.216197 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.216314 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.223047 (+ 6733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:38.223055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:38.223066 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:38.223095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:38.225905 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.226769 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.226775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.226777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.226917 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.227325 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.227328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.227330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.227794 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:38.227941 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:38.228654 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 11:23:38.228942 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":1702,"spinlock_wait_cycles":82176,"thread_start_us":1485,"threads_started":1}
I20260929 11:23:38.230012 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.205669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45051 (local address 127.0.0.1:42422)
0929 11:23:38.206615 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.206663 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:38.209476 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:38.210383 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:38.210392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:38.211245 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:38.211747 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.211773 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.213342 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.213351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.215503 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.215521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.216435 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.216445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.216661 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.217331 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:38.217367 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:38.222806 (+ 5439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:38.226039 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.226064 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.226079 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.226653 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.227018 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.227025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.227031 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.227230 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.227938 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:38.227948 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:38.228215 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 11:23:38.228498 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 11:23:38.229139 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":85,"thread_start_us":425,"threads_started":1}
I20260929 11:23:38.232939 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:23:38.234015 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.233655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45630)
0929 11:23:38.233804 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.233895 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:23:38.234323 13848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
I20260929 11:23:38.235940 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.235565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45632)
0929 11:23:38.235713 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.235783 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:23:38.236246 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36291: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 11:23:38.249364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36291
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45051,127.0.97.61:36291,127.0.97.60:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:38.285600 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } attempt: 1
I20260929 11:23:38.286772 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.286364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45648)
0929 11:23:38.286548 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.286633 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 11:23:38.287776 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.287486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45654)
0929 11:23:38.287605 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.287662 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:23:38.288024 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36291: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
I20260929 11:23:38.356325 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } attempt: 2
I20260929 11:23:38.357654 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.357170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45666)
0929 11:23:38.357404 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.357504 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 11:23:38.358862 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.358443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45672)
0929 11:23:38.358658 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.358735 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260929 11:23:38.359139 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36291: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
I20260929 11:23:38.450407 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } attempt: 3
I20260929 11:23:38.451620 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.451175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45684)
0929 11:23:38.451388 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.451480 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:23:38.452762 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.452365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45688)
0929 11:23:38.452500 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.452566 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 11:23:38.453157 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36291: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
W20260929 11:23:38.567271 13906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:38.567569 13906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:38.567749 13906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:38.575137 13906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:38.575239 13906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:38.575354 13906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:38.575448 13906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:38.575531 13906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:38.585566 13906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45051,127.0.97.61:36291,127.0.97.60:39827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:38.586933 13906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:38.588234 13906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:23:38.596405 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } attempt: 4
I20260929 11:23:38.597592 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.597095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45700)
0929 11:23:38.597377 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.597467 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 11:23:38.598722 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.598283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45706)
0929 11:23:38.598452 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.598568 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 11:23:38.598965 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36291: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36291: connect: Connection refused (error 111)
W20260929 11:23:38.600561 13911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:38.600488 13914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:38.600530 13912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:38.601639 13906 server_base.cc:1034] running on GCE node
I20260929 11:23:38.602396 13906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:38.603852 13906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:38.605093 13906 hybrid_clock.cc:648] HybridClock initialized: now 1790681018605031 us; error 51 us; skew 500 ppm
I20260929 11:23:38.605443 13906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:38 dist-test-slave-srst krb5kdc[13819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:38.608485 13906 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 11:23:38.610040 13906 webserver.cc:460] Webserver started at http://127.0.97.61:33083/ using document root <none> and password file <none>
I20260929 11:23:38.610733 13906 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:38.610885 13906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:38.611189 13906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:38.614066 13906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data/instance:
uuid: "7baf2d82697b40b3ad3e06291a9cda53"
format_stamp: "Formatted at 2026-09-29 11:23:38 on dist-test-slave-srst"
server_key: "4cd16cdba5fa625f3c4e2d92e3863cc6"
server_key_iv: "c8be164ae42b977551281410c0d3cc7d"
server_key_version: "encryptionkey@0"
I20260929 11:23:38.614835 13906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal/instance:
uuid: "7baf2d82697b40b3ad3e06291a9cda53"
format_stamp: "Formatted at 2026-09-29 11:23:38 on dist-test-slave-srst"
server_key: "4cd16cdba5fa625f3c4e2d92e3863cc6"
server_key_iv: "c8be164ae42b977551281410c0d3cc7d"
server_key_version: "encryptionkey@0"
I20260929 11:23:38.619874 13906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 11:23:38.623206 13921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:38.624441 13906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:23:38.624679 13906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal
uuid: "7baf2d82697b40b3ad3e06291a9cda53"
format_stamp: "Formatted at 2026-09-29 11:23:38 on dist-test-slave-srst"
server_key: "4cd16cdba5fa625f3c4e2d92e3863cc6"
server_key_iv: "c8be164ae42b977551281410c0d3cc7d"
server_key_version: "encryptionkey@0"
I20260929 11:23:38.624878 13906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:38.652271 13906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:38.656388 13906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:38.656701 13906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:38.675431 13906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36291
I20260929 11:23:38.675454 13972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36291 every 8 connection(s)
I20260929 11:23:38.676898 13906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
I20260929 11:23:38.680210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13906
I20260929 11:23:38.680437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-1/wal/instance
I20260929 11:23:38.680862 13973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:38.680974 388 external_mini_cluster.cc:1428] Setting key 66fb46f18fd04875166407b8c9ac16ec
I20260929 11:23:38.687206 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 29 11:23:38 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:38.709586 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.689844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.0.97.62:45051)
0929 11:23:38.690038 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:38.690042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:38.691345 (+ 1303us) server_negotiation.cc:407] Connection header received
0929 11:23:38.693173 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:38.693178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:38.693266 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:38.693373 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:38.694748 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.695451 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.697064 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.697499 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.697641 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.703280 (+ 5639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:38.703285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:38.703288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:38.703339 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:38.706102 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.707063 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.707067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.707069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.707181 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.707652 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.707655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.707657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.707984 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:38.708140 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:38.708901 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 11:23:38.709158 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:23:38.710245 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.690381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45051 (local address 127.0.0.1:42436)
0929 11:23:38.691185 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.691220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:38.692905 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:38.693596 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:38.693604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:38.694085 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:38.694590 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.694605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.695628 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.695632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.696888 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.696901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.697725 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.697732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.697962 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.698634 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:38.698656 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:38.703006 (+ 4350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:38.706245 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.706257 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.706278 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.706946 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.707283 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.707291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.707297 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.707543 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.708115 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:38.708121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:38.708444 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 11:23:38.708737 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 11:23:38.709126 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":343,"threads_started":1}
I20260929 11:23:38.711942 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:23:38 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:23:38.726336 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.712773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45712)
0929 11:23:38.712960 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.712975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:38.713060 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:38.714148 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:38.714152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:38.714578 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:38.714700 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.714708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.716027 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.716033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.716920 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.716931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.717748 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.717753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.717956 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.718548 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:38.718563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:38.720732 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:38.723826 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.723833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.723840 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.724221 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.724690 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.724698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.724702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.724868 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.725607 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:38.725611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:38.725737 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 11:23:38.725838 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 11:23:38.726083 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 11:23:38.726742 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.713095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.0.97.61:36291)
0929 11:23:38.713701 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:38.713706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:38.713767 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:23:38.713823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:38.713835 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:38.714015 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:38.714168 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:38.714844 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.715881 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.717051 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.717603 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.717743 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.720923 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:38.720933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:38.720945 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:38.720970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:38.723705 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.724352 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.724358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.724360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.724504 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.725048 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.725055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.725060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.725455 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:38.725643 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:38.726054 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 11:23:38.726246 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 11:23:38.729483 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:23:38.729925 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39827
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45051,127.0.97.61:36291,127.0.97.60:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:38.730684 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.730260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37026)
0929 11:23:38.730415 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.730556 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:23:38.730938 13924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:38.732448 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.732126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37030)
0929 11:23:38.732293 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.732361 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:23:38.732704 13973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:38.781982 13973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 1
I20260929 11:23:38.783214 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.782772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37032)
0929 11:23:38.782973 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.783071 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:23:38.784178 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.783849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37042)
0929 11:23:38.783991 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.784074 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 11:23:38.784369 13973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:38.852569 13973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 2
I20260929 11:23:38.853745 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.853291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37050)
0929 11:23:38.853485 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.853581 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 11:23:38.854667 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.854352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37054)
0929 11:23:38.854488 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.854556 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:23:38.854849 13973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:38.898315 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } attempt: 5
Sep 29 11:23:38 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:23:38.912767 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.899109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.0.97.61:36291)
0929 11:23:38.899262 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:38.899266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:38.899283 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:23:38.899461 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:38.899464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:38.899535 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:38.899614 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:38.900516 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.901187 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.902931 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.903360 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.903455 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.907467 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:38.907471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:38.907474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:38.907504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:38.910155 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.910957 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.910960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.910962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.911070 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:38.911472 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:38.911475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:38.911477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:38.911735 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:38.911836 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:38.912321 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 11:23:38.912474 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 11:23:38.913064 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.898969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45726)
0929 11:23:38.899193 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.899214 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:38.899340 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:38.899681 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:38.899686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:38.900165 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:38.900380 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.900391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.901341 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.901348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.902705 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:38.902719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:38.903526 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:38.903544 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:38.903788 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:38.904446 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:38.904474 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:38.907296 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:38.910290 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.910295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.910298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.910825 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.911171 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:38.911176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:38.911179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:38.911358 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:38.911863 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:38.911870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:38.912064 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 11:23:38.912240 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 11:23:38.912611 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 11:23:38.914152 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:23:38.915251 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.914866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37058)
0929 11:23:38.915050 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.915121 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 11:23:38.916241 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.915931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37070)
0929 11:23:38.916059 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.916139 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:23:38.916555 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:38.946015 13973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 3
I20260929 11:23:38.947150 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.946685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37076)
0929 11:23:38.946892 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.946988 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 11:23:38.948379 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.947838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37084)
0929 11:23:38.948114 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.948220 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260929 11:23:38.948596 13973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:38.967942 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 1
I20260929 11:23:38.969080 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.968658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37088)
0929 11:23:38.968847 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.968927 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:23:38.970079 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:38.969792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37102)
0929 11:23:38.969922 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:38.969985 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 11:23:38.970378 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:39.038756 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 2
I20260929 11:23:39.039940 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.039493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37116)
0929 11:23:39.039686 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.039787 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 11:23:39.040918 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.040569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37120)
0929 11:23:39.040756 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.040820 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 11:23:39.041122 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
W20260929 11:23:39.055130 13980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:39.055440 13980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:39.055594 13980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:39.062747 13980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:39.062901 13980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:39.063009 13980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:39.063118 13980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:39.063195 13980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:39.072822 13980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45051,127.0.97.61:36291,127.0.97.60:39827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:39.074126 13980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:39.075450 13980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:39.086840 13985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:39.086917 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:39.086886 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:39.087510 13980 server_base.cc:1034] running on GCE node
I20260929 11:23:39.088303 13980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:39.089761 13980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:39.090998 13980 hybrid_clock.cc:648] HybridClock initialized: now 1790681019090950 us; error 44 us; skew 500 ppm
I20260929 11:23:39.091343 13980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:23:39.091843 13973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 4
I20260929 11:23:39.092927 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.092452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37134)
0929 11:23:39.092709 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.092788 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 11:23:39.093758 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.093483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37148)
0929 11:23:39.093592 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.093667 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 11:23:39.093953 13973 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:39.097537 13980 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 11:23:39.098788 13980 webserver.cc:460] Webserver started at http://127.0.97.60:41027/ using document root <none> and password file <none>
I20260929 11:23:39.099387 13980 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:39.099474 13980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:39.099747 13980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:39.102594 13980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data/instance:
uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228"
format_stamp: "Formatted at 2026-09-29 11:23:39 on dist-test-slave-srst"
server_key: "20f8ce30072a3e1da32290e97000762a"
server_key_iv: "f63ad98570704f1178babdf7df0f05e4"
server_key_version: "encryptionkey@0"
I20260929 11:23:39.103294 13980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal/instance:
uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228"
format_stamp: "Formatted at 2026-09-29 11:23:39 on dist-test-slave-srst"
server_key: "20f8ce30072a3e1da32290e97000762a"
server_key_iv: "f63ad98570704f1178babdf7df0f05e4"
server_key_version: "encryptionkey@0"
I20260929 11:23:39.108043 13980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 11:23:39.111249 13995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:39.112466 13980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:23:39.112643 13980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal
uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228"
format_stamp: "Formatted at 2026-09-29 11:23:39 on dist-test-slave-srst"
server_key: "20f8ce30072a3e1da32290e97000762a"
server_key_iv: "f63ad98570704f1178babdf7df0f05e4"
server_key_version: "encryptionkey@0"
I20260929 11:23:39.112838 13980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:39.147337 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 3
I20260929 11:23:39.148530 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.147989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37152)
0929 11:23:39.148234 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.148335 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 11:23:39.149521 13980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:39.149642 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.149208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37158)
0929 11:23:39.149418 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.149503 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 11:23:39.149926 13899 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39827: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39827: connect: Connection refused (error 111)
I20260929 11:23:39.154093 13980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:39.154325 13980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:39.172684 13980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39827
I20260929 11:23:39.172725 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39827 every 8 connection(s)
I20260929 11:23:39.174492 13980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
I20260929 11:23:39.178313 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:39.181180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 13980
I20260929 11:23:39.181353 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790680821855275-388-0/minicluster-data/master-2/wal/instance
I20260929 11:23:39.181804 388 external_mini_cluster.cc:1428] Setting key 0ad2e41a2d0014378908bac35a2a5c00
I20260929 11:23:39.185261 14047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:39.204677 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.188707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.0.97.62:45051)
0929 11:23:39.188883 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.188887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.189614 (+ 727us) server_negotiation.cc:407] Connection header received
0929 11:23:39.189953 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.189964 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.190045 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.190154 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.191340 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.192015 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.193849 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.194521 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.194613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.199140 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.199144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.199147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.199179 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.202037 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.202698 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.202702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.202704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.202846 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.203229 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.203234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.203241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.203465 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.203620 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.204116 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.204333 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:23:39.205679 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.188767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45051 (local address 127.0.0.1:42444)
0929 11:23:39.189442 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.189479 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:39.189742 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:39.190232 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:39.190238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:39.190951 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:39.191198 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.191206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.192169 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.192174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.193675 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.193686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.195592 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.195597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.195725 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.196336 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:39.196364 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:39.198942 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:39.202169 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.202179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.202189 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.202569 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.202952 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.202958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.202961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.203109 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.203586 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:39.203595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:39.203797 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 11:23:39.204019 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 11:23:39.204769 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":323,"thread_start_us":371,"threads_started":1}
I20260929 11:23:39.205698 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.183538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.0.97.60:39827)
0929 11:23:39.184299 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.184310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.184350 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:23:39.185979 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.186001 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.186415 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.186780 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.188540 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.190548 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.192514 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.193523 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.193697 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.198759 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.198770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.198786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.198827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.201711 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.202396 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.202411 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.202414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.202587 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.202984 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.202991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.202994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.203451 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.203748 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.204053 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.204643 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":562,"thread_start_us":348,"threads_started":1}
I20260929 11:23:39.207116 14047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:39.220845 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.207794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36291 (local address 127.0.0.1:45732)
0929 11:23:39.207964 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.207982 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:39.208076 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:39.208812 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:39.208816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:39.209289 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:39.209468 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.209479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.210428 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.210433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.211374 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.211385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.212118 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.212124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.212331 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.212874 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:39.212901 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:39.215027 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:39.218445 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.218451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.218456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.218868 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.219282 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.219285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.219287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.219462 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.220060 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:39.220064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:39.220190 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 11:23:39.220319 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 11:23:39.220488 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:23:39.221091 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.207971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.0.97.61:36291)
0929 11:23:39.208182 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.208188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.208207 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:23:39.208538 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.208543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.208656 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.208756 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.209611 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.210280 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.211549 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.211981 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.212071 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.215208 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.215212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.215215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.215245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.218307 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.218976 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.218981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.218985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.219171 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.219573 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.219578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.219582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.219920 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.220082 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.220450 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.220683 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:23:39.221985 14047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:23:39.224238 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.208767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.0.97.62:45051)
0929 11:23:39.208932 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.208935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.209066 (+ 131us) server_negotiation.cc:407] Connection header received
0929 11:23:39.209247 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.209250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.209300 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.209365 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.210671 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.211390 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.213009 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.213384 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.213476 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.217803 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.217812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.217819 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.217858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.221329 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.222018 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.222023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.222026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.222155 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.222524 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.222531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.222538 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.222789 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.223556 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.223689 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.223920 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:23:39.235807 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.222993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.0.97.60:39827)
0929 11:23:39.223167 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.223171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.223182 (+ 11us) server_negotiation.cc:407] Connection header received
0929 11:23:39.223213 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.223216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.223259 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.223355 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.224112 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.224960 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.226278 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.227036 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.227168 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.230231 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.230237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.230242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.230277 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.232996 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.233792 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.233796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.233798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.233910 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.234280 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.234288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.234292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.234619 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.234771 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.235156 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.235506 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":35}
I20260929 11:23:39.235806 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.222770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37170)
0929 11:23:39.222943 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.222961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:39.223047 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:39.223369 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:39.223373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:39.223822 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:39.223977 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.223988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.225093 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.225098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.226114 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.226126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.227178 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.227182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.227311 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.227887 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:39.227903 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:39.230034 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:39.233143 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.233149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.233154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.233654 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.234022 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.234027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.234031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.234170 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.234736 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:39.234740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:39.234877 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 11:23:39.235038 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 11:23:39.235361 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 11:23:39.240971 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: Bootstrap starting.
I20260929 11:23:39.242839 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.225827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.61:36291)
0929 11:23:39.225993 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.225999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.226017 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:23:39.226101 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.226105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.226175 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.226269 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.227712 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.228507 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.229909 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.230463 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.230599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.235231 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.235236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.235239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.235268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.240289 (+ 5021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.241155 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.241173 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.241175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.241296 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.241690 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.241693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.241695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.241935 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.242137 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.242454 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.242613 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:23:39.244884 14047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:39.245973 14047 log.cc:824] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:39.247449 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:23:39.248548 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: No bootstrap required, opened a new log
I20260929 11:23:39.261303 14047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.261907 14047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:39.262095 14047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7fcfcf179a4cc19ba2d79b9ebc6228, State: Initialized, Role: FOLLOWER
I20260929 11:23:39.262969 14047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.269176 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.269659 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.270884 14047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:39.279959 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.251937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.0.97.62:45051)
0929 11:23:39.252125 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.252132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.259842 (+ 7710us) server_negotiation.cc:407] Connection header received
0929 11:23:39.260088 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.260094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.260187 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.260264 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.262659 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.263775 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.266563 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.267252 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.267356 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.272424 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.272431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.272436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.272479 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.276640 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.277454 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.277462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.277465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.277613 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.278123 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.278131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.278137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.278490 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.278738 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.279353 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.279585 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:23:39.279964 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.252524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.0.97.60:39827)
0929 11:23:39.256527 (+ 4003us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.256536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.259527 (+ 2991us) server_negotiation.cc:407] Connection header received
0929 11:23:39.259790 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.259798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.259890 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.259987 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.262658 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.263775 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.266574 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.267276 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.267358 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.271917 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.271924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.271929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.271976 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.275740 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.276676 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.276684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.276691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.276876 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.277407 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.277415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.277422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.278549 (+ 1127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.278768 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.279353 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.279591 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3842}
I20260929 11:23:39.281523 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.252383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.0.97.61:36291)
0929 11:23:39.258203 (+ 5820us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.258211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.259781 (+ 1570us) server_negotiation.cc:407] Connection header received
0929 11:23:39.259844 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.259849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.259922 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.260011 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.262719 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.263890 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.267195 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.268087 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.268220 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.272097 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.272103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.272109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.272156 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.277134 (+ 4978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.278253 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.278260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.278311 (+ 51us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.278558 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.279098 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.279102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.279105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.279543 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.279859 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.280709 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.281016 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5670}
W20260929 11:23:39.293759 14071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:23:39.293972 14071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:23:39.290778 13980 master_runner.cc:186] Error getting master registration for 127.0.97.62:45051: OK, instance_id { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" instance_seqno: 1790681018185162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:23:39.300185 13980 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:23:39.327131 13899 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 4
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 11:23:39.342272 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.327750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.0.97.60:39827)
0929 11:23:39.327927 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.327931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.327990 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:23:39.328161 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.328164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.328246 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.328360 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.329324 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.330186 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.331926 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.332351 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.332437 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.336823 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.336828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.336831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.336861 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.339636 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.340237 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.340244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.340251 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.340411 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.340861 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.340866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.340869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.341125 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.341292 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.341628 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.341876 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:23:39.342290 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.327664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37196)
0929 11:23:39.327903 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.327925 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:39.328050 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:39.328423 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:39.328429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:39.328974 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:39.329188 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.329201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.330318 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.330324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.331791 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.331806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.332491 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.332498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.332733 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.333350 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:39.333373 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:39.336632 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:39.339770 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.339775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.339779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.340115 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.340523 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.340529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.340532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.340755 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.341248 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:39.341252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:39.341397 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 11:23:39.341533 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:23:39.341890 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 11:23:39.347584 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: Bootstrap starting.
I20260929 11:23:39.350807 13899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:39.351794 13899 log.cc:824] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:39.354370 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: No bootstrap required, opened a new log
I20260929 11:23:39.362493 13899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.362751 13899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:39.362887 13899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0699265d4c4f4703be44e9b89aed6df1, State: Initialized, Role: FOLLOWER
I20260929 11:23:39.363415 13899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.364293 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.364689 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.365438 13899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:23:39.377390 14085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:23:39.377588 14085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:23:39.374989 13832 master_runner.cc:186] Error getting master registration for 127.0.97.61:36291: OK, instance_id { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" instance_seqno: 1790681018670123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:23:39.386274 13832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:23:39.393191 13973 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } attempt: 5
Sep 29 11:23:39 dist-test-slave-srst krb5kdc[13819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260929 11:23:39.406301 14001 tablet.cc:2369] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:23:39.409947 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.393768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39827 (local address 127.0.0.1:37206)
0929 11:23:39.394012 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:39.394032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:39.394150 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:39.394510 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:39.394515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:39.395029 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:39.395265 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.395278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.396774 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.396780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.398144 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:39.398159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.399112 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.399119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:39.399336 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.400005 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:39.400050 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:39.402773 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:39.406962 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.406967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.406971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.407444 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.407868 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:39.407873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:39.407877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:39.408047 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:39.408827 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:39.408901 (+ 74us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:39.409062 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 11:23:39.409216 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 11:23:39.409495 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 11:23:39.411010 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:39.393829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.97.60:39827)
0929 11:23:39.394003 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:39.394010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:39.394108 (+ 98us) server_negotiation.cc:407] Connection header received
0929 11:23:39.394270 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:39.394275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:39.394363 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:39.394464 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:39.395414 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.396619 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.398287 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:39.398957 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:39.399080 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:39.402969 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:39.402976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:39.402981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:39.403020 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:39.406807 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.407581 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.407586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.407593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.407737 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:39.408178 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:39.408182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:39.408186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:39.408582 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:39.408763 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:39.410401 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0929 11:23:39.410617 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:23:39.416842 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53: Bootstrap starting.
I20260929 11:23:39.420840 13973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:39.422082 13973 log.cc:824] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53: Log is configured to *not* fsync() on all Append() calls
W20260929 11:23:39.424470 13853 tablet.cc:2369] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:23:39.425334 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53: No bootstrap required, opened a new log
I20260929 11:23:39.438025 13973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.438405 13973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:39.438525 13973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7baf2d82697b40b3ad3e06291a9cda53, State: Initialized, Role: FOLLOWER
I20260929 11:23:39.439198 13973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.440338 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.440855 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.441587 13973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:39.460335 13906 master_runner.cc:263] No leader master found from master 7baf2d82697b40b3ad3e06291a9cda53
I20260929 11:23:39.460466 13906 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 11:23:39.471227 14099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:23:39.471464 14099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:23:39.503415 14088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:23:39.503755 14088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.505452 14088 leader_election.cc:288] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0699265d4c4f4703be44e9b89aed6df1 (127.0.97.62:45051), 5e7fcfcf179a4cc19ba2d79b9ebc6228 (127.0.97.60:39827)
I20260929 11:23:39.506695 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7baf2d82697b40b3ad3e06291a9cda53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0699265d4c4f4703be44e9b89aed6df1" is_pre_election: true
I20260929 11:23:39.507294 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7baf2d82697b40b3ad3e06291a9cda53 in term 0.
I20260929 11:23:39.508049 13923 leader_election.cc:302] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0699265d4c4f4703be44e9b89aed6df1, 7baf2d82697b40b3ad3e06291a9cda53; no voters:
I20260929 11:23:39.508960 14088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:23:39.509109 14088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:23:39.509210 14088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:39.509222 14022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7baf2d82697b40b3ad3e06291a9cda53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" is_pre_election: true
I20260929 11:23:39.509825 14022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7baf2d82697b40b3ad3e06291a9cda53 in term 0.
I20260929 11:23:39.512804 14088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.516285 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7baf2d82697b40b3ad3e06291a9cda53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0699265d4c4f4703be44e9b89aed6df1"
I20260929 11:23:39.516489 13874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:39.519850 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7baf2d82697b40b3ad3e06291a9cda53 in term 1.
I20260929 11:23:39.520315 13923 leader_election.cc:302] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0699265d4c4f4703be44e9b89aed6df1, 7baf2d82697b40b3ad3e06291a9cda53; no voters:
I20260929 11:23:39.525406 14022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7baf2d82697b40b3ad3e06291a9cda53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228"
I20260929 11:23:39.525676 14022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:39.526508 14088 leader_election.cc:288] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [CANDIDATE]: Term 1 election: Requested vote from peers 0699265d4c4f4703be44e9b89aed6df1 (127.0.97.62:45051), 5e7fcfcf179a4cc19ba2d79b9ebc6228 (127.0.97.60:39827)
I20260929 11:23:39.529482 14101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:39.529565 14022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7baf2d82697b40b3ad3e06291a9cda53 in term 1.
I20260929 11:23:39.529687 14101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [term 1 LEADER]: Becoming Leader. State: Replica: 7baf2d82697b40b3ad3e06291a9cda53, State: Running, Role: LEADER
I20260929 11:23:39.530326 14101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } }
I20260929 11:23:39.543356 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7baf2d82697b40b3ad3e06291a9cda53. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.543670 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:39.555511 14103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:39.557732 14103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:39.577070 14022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [term 1 FOLLOWER]: Refusing update from remote peer 7baf2d82697b40b3ad3e06291a9cda53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:39.577678 13874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [term 1 FOLLOWER]: Refusing update from remote peer 7baf2d82697b40b3ad3e06291a9cda53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:39.578477 14101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:39.579176 14088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:39.613582 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7baf2d82697b40b3ad3e06291a9cda53. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.613935 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.619951 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7baf2d82697b40b3ad3e06291a9cda53. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.621101 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.625361 14106 mvcc.cc:205] Tried to move back new op lower bound from 7334629456176107520 to 7334629456001748992. Current Snapshot: MvccSnapshot[applied={T|T < 7334629456176107520}]
I20260929 11:23:39.629091 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.629328 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.630985 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.631313 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:39.631632 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.631893 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:39.632480 14103 catalog_manager.cc:1347] Generated new cluster ID: c5291538f4274545bfec1511443382b0
I20260929 11:23:39.632579 14103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:23:39.633709 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7baf2d82697b40b3ad3e06291a9cda53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0699265d4c4f4703be44e9b89aed6df1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45051 } } peers { permanent_uuid: "7baf2d82697b40b3ad3e06291a9cda53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36291 } } peers { permanent_uuid: "5e7fcfcf179a4cc19ba2d79b9ebc6228" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39827 } } }
I20260929 11:23:39.633929 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:23:39.660713 14103 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:23:39.662863 14103 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:23:39.690349 14103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7baf2d82697b40b3ad3e06291a9cda53: Generated new TSK 0
I20260929 11:23:39.691521 14103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:23:40.295877 14071 catalog_manager.cc:1259] Loaded cluster ID: c5291538f4274545bfec1511443382b0
I20260929 11:23:40.296015 14071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: loading cluster ID for follower catalog manager: success
I20260929 11:23:40.298399 14071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: acquiring CA information for follower catalog manager: success
I20260929 11:23:40.300103 14071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e7fcfcf179a4cc19ba2d79b9ebc6228: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:23:40.379598 14085 catalog_manager.cc:1259] Loaded cluster ID: c5291538f4274545bfec1511443382b0
I20260929 11:23:40.379729 14085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: loading cluster ID for follower catalog manager: success
I20260929 11:23:40.382217 14085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: acquiring CA information for follower catalog manager: success
I20260929 11:23:40.383847 14085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0699265d4c4f4703be44e9b89aed6df1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:23:40.837062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13832
I20260929 11:23:40.838316 13894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:40.950234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13832
I20260929 11:23:40.970892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13906
I20260929 11:23:40.972066 13968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:41.093518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13906
I20260929 11:23:41.113559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 13980
I20260929 11:23:41.114647 14042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:41.221258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 13980
2026-09-29T11:23:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4744 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:23:41 dist-test-slave-srst krb5kdc[14121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:23:41 dist-test-slave-srst krb5kdc[14121](info): set up 2 sockets
Sep 29 11:23:41 dist-test-slave-srst krb5kdc[14121](info): commencing operation
krb5kdc: starting...
W20260929 11:23:42.313282 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:23:42 dist-test-slave-srst krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:23:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:23:42.535923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34207
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:42.856634 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:42.857012 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:42.857170 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:42.865037 14134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:42.865193 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:42.865294 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:42.865396 14134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:42.865470 14134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:42.875412 14134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:42.876865 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:42.878176 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:42.890050 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:42.890007 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:42.890105 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:42.891249 14134 server_base.cc:1034] running on GCE node
I20260929 11:23:42.891929 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:42.893318 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:42.894528 14134 hybrid_clock.cc:648] HybridClock initialized: now 1790681022894486 us; error 40 us; skew 500 ppm
I20260929 11:23:42.894806 14134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:42 dist-test-slave-srst krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:42.897770 14134 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:42.899124 14134 webserver.cc:460] Webserver started at http://127.0.97.62:40363/ using document root <none> and password file <none>
I20260929 11:23:42.899730 14134 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:42.899824 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:42.900080 14134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:42.902980 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "c6e626690011472ea9a854f57a50b9dd"
format_stamp: "Formatted at 2026-09-29 11:23:42 on dist-test-slave-srst"
server_key: "dc1a02ebfb1a7f455bd970c6924a60d7"
server_key_iv: "16661fdd193045cc65738b41259eaf0d"
server_key_version: "encryptionkey@0"
I20260929 11:23:42.903668 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "c6e626690011472ea9a854f57a50b9dd"
format_stamp: "Formatted at 2026-09-29 11:23:42 on dist-test-slave-srst"
server_key: "dc1a02ebfb1a7f455bd970c6924a60d7"
server_key_iv: "16661fdd193045cc65738b41259eaf0d"
server_key_version: "encryptionkey@0"
I20260929 11:23:42.908823 14134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 11:23:42.912145 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:42.913515 14134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:23:42.913676 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "c6e626690011472ea9a854f57a50b9dd"
format_stamp: "Formatted at 2026-09-29 11:23:42 on dist-test-slave-srst"
server_key: "dc1a02ebfb1a7f455bd970c6924a60d7"
server_key_iv: "16661fdd193045cc65738b41259eaf0d"
server_key_version: "encryptionkey@0"
I20260929 11:23:42.913851 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:42.943393 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:42.947933 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:42.948182 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:42.966292 14134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34207
I20260929 11:23:42.966323 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34207 every 8 connection(s)
I20260929 11:23:42.967689 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:42.971292 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:42.977075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14134
I20260929 11:23:42.977289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:23:42.977891 388 external_mini_cluster.cc:1428] Setting key f63028c1d130556f71f35aecb8604afd
I20260929 11:23:42.981769 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd: Bootstrap starting.
I20260929 11:23:42.985390 14201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:42.986441 14201 log.cc:824] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:42.989293 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd: No bootstrap required, opened a new log
Sep 29 11:23:42 dist-test-slave-srst krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:42.998641 14201 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e626690011472ea9a854f57a50b9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34207 } }
I20260929 11:23:42.999070 14201 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:42.999182 14201 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e626690011472ea9a854f57a50b9dd, State: Initialized, Role: FOLLOWER
I20260929 11:23:42.999795 14201 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e626690011472ea9a854f57a50b9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34207 } }
I20260929 11:23:43.000104 14201 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:23:43.000289 14201 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:23:43.000465 14201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:43.001598 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:42.979664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.0.97.62:34207)
0929 11:23:42.980455 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:42.980466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:42.980506 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:23:42.982311 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:42.982332 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:42.982686 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:42.983008 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:42.984816 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:42.986193 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:42.987887 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:42.988522 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:42.988723 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:42.992850 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:42.992860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:42.992883 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:42.992931 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:42.996915 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:42.997596 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:42.997604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:42.997613 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:42.997824 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:42.998181 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:42.998185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:42.998188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:42.998844 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:42.999145 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:42.999658 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 11:23:43.000080 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":387,"threads_started":1}
I20260929 11:23:43.003609 14201 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e626690011472ea9a854f57a50b9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34207 } }
I20260929 11:23:43.004204 14201 leader_election.cc:302] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6e626690011472ea9a854f57a50b9dd; no voters:
I20260929 11:23:43.004920 14201 leader_election.cc:288] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:23:43.005021 14206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:43.005941 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:23:43.006281 14206 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [term 1 LEADER]: Becoming Leader. State: Replica: c6e626690011472ea9a854f57a50b9dd, State: Running, Role: LEADER
I20260929 11:23:43.006827 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e626690011472ea9a854f57a50b9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34207 } }
I20260929 11:23:43.007189 14201 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:43.009783 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6e626690011472ea9a854f57a50b9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e626690011472ea9a854f57a50b9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34207 } } }
I20260929 11:23:43.010238 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:43.012905 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6e626690011472ea9a854f57a50b9dd. Latest consensus state: current_term: 1 leader_uuid: "c6e626690011472ea9a854f57a50b9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6e626690011472ea9a854f57a50b9dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34207 } } }
I20260929 11:23:43.013319 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:43.013893 14211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:43.018774 14211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:43.027349 14211 catalog_manager.cc:1347] Generated new cluster ID: dc2da8b20318418a98528de1599f7b3c
I20260929 11:23:43.027459 14211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:23:43.042526 14211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:23:43.044399 14211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:23:43.052558 14211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6e626690011472ea9a854f57a50b9dd: Generated new TSK 0
I20260929 11:23:43.053607 14211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:23:43.066051 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14134
I20260929 11:23:43.067130 14196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:43.157372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14134
2026-09-29T11:23:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1935 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:23:43 dist-test-slave-srst krb5kdc[14229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:23:43 dist-test-slave-srst krb5kdc[14229](info): set up 2 sockets
Sep 29 11:23:43 dist-test-slave-srst krb5kdc[14229](info): commencing operation
krb5kdc: starting...
W20260929 11:23:44.245069 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:23:44 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:23:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:23:44.468451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38265
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:44.785224 14242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:44.785494 14242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:44.785581 14242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:44.792629 14242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:44.792725 14242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:44.792769 14242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:44.792822 14242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:44.792857 14242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:44.802124 14242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:44.803356 14242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:44.804651 14242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:44.816814 14250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:44.816857 14247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:44.816893 14248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:44.817158 14242 server_base.cc:1034] running on GCE node
I20260929 11:23:44.818043 14242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:44.819468 14242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:44.820734 14242 hybrid_clock.cc:648] HybridClock initialized: now 1790681024820671 us; error 38 us; skew 500 ppm
I20260929 11:23:44.821090 14242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:44 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:44.828555 14242 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:44.830206 14242 webserver.cc:460] Webserver started at http://127.0.97.62:36275/ using document root <none> and password file <none>
I20260929 11:23:44.831458 14242 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:44.831622 14242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:44.831956 14242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:44.835075 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "f040e582a6a5490f9e25fd428ba022e8"
format_stamp: "Formatted at 2026-09-29 11:23:44 on dist-test-slave-srst"
server_key: "904d6887dffeb784350f5c95c22ab60d"
server_key_iv: "f158138d1e16592449b2ecd1cfe1e44e"
server_key_version: "encryptionkey@0"
I20260929 11:23:44.835870 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "f040e582a6a5490f9e25fd428ba022e8"
format_stamp: "Formatted at 2026-09-29 11:23:44 on dist-test-slave-srst"
server_key: "904d6887dffeb784350f5c95c22ab60d"
server_key_iv: "f158138d1e16592449b2ecd1cfe1e44e"
server_key_version: "encryptionkey@0"
I20260929 11:23:44.841267 14242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:23:44.844753 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:44.846061 14242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:23:44.846359 14242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "f040e582a6a5490f9e25fd428ba022e8"
format_stamp: "Formatted at 2026-09-29 11:23:44 on dist-test-slave-srst"
server_key: "904d6887dffeb784350f5c95c22ab60d"
server_key_iv: "f158138d1e16592449b2ecd1cfe1e44e"
server_key_version: "encryptionkey@0"
I20260929 11:23:44.846593 14242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:44.892023 14242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:44.896061 14242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:44.896385 14242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:44.914623 14242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38265
I20260929 11:23:44.914661 14308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38265 every 8 connection(s)
I20260929 11:23:44.915927 14242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:44.919544 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:44.920146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14242
I20260929 11:23:44.920372 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:23:44.921064 388 external_mini_cluster.cc:1428] Setting key ba6742adf5d49dae1f2576bfe8009c27
I20260929 11:23:44.933274 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap starting.
Sep 29 11:23:44 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:44.936810 14309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:44.937825 14309 log.cc:824] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:44.941020 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: No bootstrap required, opened a new log
I20260929 11:23:44.944568 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:44.922753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.97.62:38265)
0929 11:23:44.923485 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:44.923499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:44.923539 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:23:44.925635 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:44.925656 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:44.926034 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:44.926381 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:44.928112 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:44.929381 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:44.931224 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:44.931858 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:44.931977 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:44.936518 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:44.936525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:44.936539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:44.936572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:44.940270 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:44.940999 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:44.941014 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:44.941022 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:44.941209 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:44.941570 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:44.941574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:44.941577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:44.942239 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:44.942514 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:44.942830 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:23:44.943245 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":357,"threads_started":1}
I20260929 11:23:44.950368 14309 raft_consensus.cc:348] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:44.950788 14309 raft_consensus.cc:374] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:44.950896 14309 raft_consensus.cc:729] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f040e582a6a5490f9e25fd428ba022e8, State: Initialized, Role: FOLLOWER
I20260929 11:23:44.951454 14309 consensus_queue.cc:260] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:44.951690 14309 raft_consensus.cc:388] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:23:44.951830 14309 raft_consensus.cc:482] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:23:44.952003 14309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:44.954140 14309 raft_consensus.cc:504] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:44.954710 14309 leader_election.cc:302] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f040e582a6a5490f9e25fd428ba022e8; no voters:
I20260929 11:23:44.955339 14309 leader_election.cc:288] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:23:44.955435 14314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:44.956091 14314 raft_consensus.cc:686] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 LEADER]: Becoming Leader. State: Replica: f040e582a6a5490f9e25fd428ba022e8, State: Running, Role: LEADER
I20260929 11:23:44.956722 14314 consensus_queue.cc:237] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:44.957376 14309 sys_catalog.cc:564] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:44.959556 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f040e582a6a5490f9e25fd428ba022e8. Latest consensus state: current_term: 1 leader_uuid: "f040e582a6a5490f9e25fd428ba022e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } } }
I20260929 11:23:44.959919 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:44.959550 14315 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f040e582a6a5490f9e25fd428ba022e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } } }
I20260929 11:23:44.960029 14315 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:44.962123 14321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:44.967090 14321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:44.975651 14321 catalog_manager.cc:1347] Generated new cluster ID: 6336938a83794aec927b3d87caa0cda7
I20260929 11:23:44.975771 14321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:23:44.985579 14321 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:23:44.986815 14321 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:23:45.000720 14321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Generated new TSK 0
I20260929 11:23:45.001531 14321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:23:45.067891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38265
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:45.410898 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:45.411167 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:45.411321 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:45.418533 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:45.418632 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:45.418731 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:23:45.427735 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:45.428946 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:45.430136 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:45.442801 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:45.442761 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:45.442845 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:45.444092 14335 server_base.cc:1034] running on GCE node
I20260929 11:23:45.444805 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:45.445741 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:45.447001 14335 hybrid_clock.cc:648] HybridClock initialized: now 1790681025446953 us; error 55 us; skew 500 ppm
I20260929 11:23:45.447327 14335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:45 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:45.450551 14335 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:23:45.451982 14335 webserver.cc:460] Webserver started at http://127.0.97.1:44365/ using document root <none> and password file <none>
I20260929 11:23:45.452751 14335 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:45.452911 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:45.453204 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:45.456161 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "16564e3d08c94134af84fc187db0a520"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "9e59de2633192efce0db75aae2b7934f"
server_key_iv: "6ba4277e72d7b670520592508d4b9dc3"
server_key_version: "encryptionkey@0"
I20260929 11:23:45.456995 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "16564e3d08c94134af84fc187db0a520"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "9e59de2633192efce0db75aae2b7934f"
server_key_iv: "6ba4277e72d7b670520592508d4b9dc3"
server_key_version: "encryptionkey@0"
I20260929 11:23:45.462602 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:23:45.466025 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:45.467248 14335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:23:45.467437 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "16564e3d08c94134af84fc187db0a520"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "9e59de2633192efce0db75aae2b7934f"
server_key_iv: "6ba4277e72d7b670520592508d4b9dc3"
server_key_version: "encryptionkey@0"
I20260929 11:23:45.467658 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:45.484262 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:45.488072 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:45.488306 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:45.489282 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:45.490518 14335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:45.490623 14335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:45.490711 14335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:45.490756 14335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:45.523312 14335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42713
I20260929 11:23:45.523357 14462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42713 every 8 connection(s)
I20260929 11:23:45.524549 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:23:45.533406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14335
I20260929 11:23:45.533627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:23:45.534430 388 external_mini_cluster.cc:1428] Setting key b473f40c193304d6caf15f80c89db965
Sep 29 11:23:45 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:45.546116 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:45.527127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53327 (local address 127.0.97.62:38265)
0929 11:23:45.527697 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:45.527703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:45.528751 (+ 1048us) server_negotiation.cc:407] Connection header received
0929 11:23:45.530427 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:45.530431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:45.530494 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:45.530625 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:45.532098 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:45.532959 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:45.535281 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:45.535846 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:45.535974 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:45.539486 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:45.539494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:45.539501 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:45.539562 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:45.543111 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:45.543823 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:45.543829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:45.543832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:45.543963 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:45.544431 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:45.544434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:45.544436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:45.544755 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:45.544870 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:45.545703 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 11:23:45.545869 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":308,"threads_started":1}
I20260929 11:23:45.547623 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:45.527820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:53327)
0929 11:23:45.528524 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:45.528562 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:45.530186 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:45.530833 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:45.530841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:45.531362 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:45.531942 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:45.531960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:45.533108 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:45.533115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:45.535114 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:45.535127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:45.536181 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:45.536196 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:45.536443 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:45.537143 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:45.537164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:45.539270 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:45.543250 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:45.543256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:45.543271 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:45.543686 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:45.544069 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:45.544076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:45.544080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:45.544320 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:45.544881 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:45.544890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:45.545327 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 11:23:45.545627 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 11:23:45.546178 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
I20260929 11:23:45.549219 14463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38265
I20260929 11:23:45.549543 14463 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:45.550357 14463 heartbeater.cc:511] Master 127.0.97.62:38265 requested a full tablet report, sending...
I20260929 11:23:45.552568 14274 ts_manager.cc:194] Registered new tserver with Master: 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713)
I20260929 11:23:45.554100 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53327
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:23:45.596693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38265
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:45.921155 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:45.921465 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:45.921628 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:45.932467 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:45.932603 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:45.932782 14469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:23:45.943236 14469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:45.944357 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:45.945637 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:45.958112 14474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:45.958119 14477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:45.958220 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:45.959518 14469 server_base.cc:1034] running on GCE node
I20260929 11:23:45.960075 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:45.961004 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:45.962246 14469 hybrid_clock.cc:648] HybridClock initialized: now 1790681025962197 us; error 46 us; skew 500 ppm
I20260929 11:23:45.962550 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:45 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:45.965620 14469 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:23:45.967005 14469 webserver.cc:460] Webserver started at http://127.0.97.2:38899/ using document root <none> and password file <none>
I20260929 11:23:45.967660 14469 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:45.967787 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:45.968113 14469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:45.970953 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "2d50dc3297ef492e9ef7662bba3a63fc"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "688bac4cd5633f0302e07bd4afa12e8a"
server_key_iv: "e34b60573c0c7bd53eec618c6730fd8a"
server_key_version: "encryptionkey@0"
I20260929 11:23:45.971846 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2d50dc3297ef492e9ef7662bba3a63fc"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "688bac4cd5633f0302e07bd4afa12e8a"
server_key_iv: "e34b60573c0c7bd53eec618c6730fd8a"
server_key_version: "encryptionkey@0"
I20260929 11:23:45.976763 14469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:23:45.979923 14484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:45.981204 14469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:23:45.981353 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "2d50dc3297ef492e9ef7662bba3a63fc"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "688bac4cd5633f0302e07bd4afa12e8a"
server_key_iv: "e34b60573c0c7bd53eec618c6730fd8a"
server_key_version: "encryptionkey@0"
I20260929 11:23:45.981540 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:45.996078 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:46.000181 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:46.000455 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:46.001631 14469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:46.003250 14469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:46.003352 14469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:46.003470 14469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:46.003556 14469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:46.037065 14469 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33379
I20260929 11:23:46.037109 14596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33379 every 8 connection(s)
I20260929 11:23:46.038543 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:23:46.048039 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14469
I20260929 11:23:46.048229 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:23:46.048799 388 external_mini_cluster.cc:1428] Setting key 42a18666ff49152928ca51fe858b04a0
Sep 29 11:23:46 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:46.058861 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.041135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50529 (local address 127.0.97.62:38265)
0929 11:23:46.041319 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.041324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.042478 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 11:23:46.044312 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.044318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.044416 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.044531 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:46.046028 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.046751 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.048327 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.048968 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.049107 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.052997 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:46.053002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:46.053005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:46.053036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:46.055735 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.056441 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.056447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.056451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.056625 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.057060 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.057067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.057071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.057419 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:46.057585 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.058279 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.058519 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:23:46.059906 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.041617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:50529)
0929 11:23:46.042310 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.042349 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.044014 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.044777 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.044786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.045286 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:46.045843 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.045859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.046899 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.046906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.048178 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.048201 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.049191 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.049199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.049512 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.050331 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:46.050367 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:46.052770 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:46.055885 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:46.055894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:46.055911 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:46.056310 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:46.056746 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:46.056751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:46.056755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:46.056950 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:46.057572 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:46.057579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:46.057920 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.058175 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.058765 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260929 11:23:46.061493 14597 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38265
I20260929 11:23:46.061825 14597 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:46.062687 14597 heartbeater.cc:511] Master 127.0.97.62:38265 requested a full tablet report, sending...
I20260929 11:23:46.064319 14274 ts_manager.cc:194] Registered new tserver with Master: 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2:33379)
I20260929 11:23:46.065604 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50529
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:23:46.098006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38265
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:46.408576 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:46.408841 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:46.409101 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:46.416334 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:46.416436 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:46.416692 14602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:23:46.426091 14602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:46.427280 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:46.428570 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:46.441874 14607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:46.441819 14608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:46.441829 14610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:46.442924 14602 server_base.cc:1034] running on GCE node
I20260929 11:23:46.443579 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:46.444636 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:46.445888 14602 hybrid_clock.cc:648] HybridClock initialized: now 1790681026445844 us; error 38 us; skew 500 ppm
I20260929 11:23:46.446213 14602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:46 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:46.449318 14602 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:23:46.450784 14602 webserver.cc:460] Webserver started at http://127.0.97.3:33617/ using document root <none> and password file <none>
I20260929 11:23:46.451462 14602 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:46.451563 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:46.451800 14602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:23:46.454782 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "836d7249ccc94daaa9432c1fc7b766bd"
format_stamp: "Formatted at 2026-09-29 11:23:46 on dist-test-slave-srst"
server_key: "6857097f065968a883e65011dcdd3ae5"
server_key_iv: "5497cc979807cc4bdca528a75a6d5546"
server_key_version: "encryptionkey@0"
I20260929 11:23:46.455499 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "836d7249ccc94daaa9432c1fc7b766bd"
format_stamp: "Formatted at 2026-09-29 11:23:46 on dist-test-slave-srst"
server_key: "6857097f065968a883e65011dcdd3ae5"
server_key_iv: "5497cc979807cc4bdca528a75a6d5546"
server_key_version: "encryptionkey@0"
I20260929 11:23:46.460630 14602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 11:23:46.464090 14617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:46.465400 14602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:23:46.465528 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "836d7249ccc94daaa9432c1fc7b766bd"
format_stamp: "Formatted at 2026-09-29 11:23:46 on dist-test-slave-srst"
server_key: "6857097f065968a883e65011dcdd3ae5"
server_key_iv: "5497cc979807cc4bdca528a75a6d5546"
server_key_version: "encryptionkey@0"
I20260929 11:23:46.465692 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:46.477882 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:46.481829 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:46.482074 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:46.482908 14602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:46.484076 14602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:46.484143 14602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:46.484220 14602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:46.484261 14602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:46.515177 14602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35525
I20260929 11:23:46.515264 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35525 every 8 connection(s)
I20260929 11:23:46.516429 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:23:46.517269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14602
I20260929 11:23:46.517452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:23:46.518009 388 external_mini_cluster.cc:1428] Setting key 427d23552c734282a9cc7a3bf6f710cf
Sep 29 11:23:46 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:46.536659 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.519217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46899 (local address 127.0.97.62:38265)
0929 11:23:46.519397 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.519402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.520936 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 11:23:46.522850 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.522856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.522946 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.523058 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:46.524486 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.525295 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.526860 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.527254 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.527386 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.530649 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:46.530653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:46.530657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:46.530698 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:46.533751 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.534425 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.534430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.534434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.534581 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.534998 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.535003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.535006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.535274 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:46.535427 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.536084 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.536293 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:23:46.537535 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.519926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:46899)
0929 11:23:46.520796 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.520839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.522581 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.523289 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.523298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.523783 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:46.524321 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.524337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.525441 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.525446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.526665 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.526681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.527495 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.527502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.527723 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.528308 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:46.528329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:46.530426 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:46.533891 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:46.533900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:46.533915 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:46.534298 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:46.534692 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:46.534696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:46.534700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:46.534876 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:46.535400 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:46.535407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:46.535715 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.535970 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.536460 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":304,"threads_started":1}
I20260929 11:23:46.538741 14730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38265
I20260929 11:23:46.539032 14730 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:46.539716 14730 heartbeater.cc:511] Master 127.0.97.62:38265 requested a full tablet report, sending...
I20260929 11:23:46.541091 14274 ts_manager.cc:194] Registered new tserver with Master: 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525)
I20260929 11:23:46.541898 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46899
I20260929 11:23:46.545684 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:23:46.557848 14463 heartbeater.cc:503] Master 127.0.97.62:38265 was elected leader, sending a full tablet report...
I20260929 11:23:46.562875 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.550024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.0.97.62:38265)
0929 11:23:46.550188 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.550193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.550584 (+ 391us) server_negotiation.cc:407] Connection header received
0929 11:23:46.550778 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.550785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.550847 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.550948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:46.552339 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.553128 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.554393 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.554926 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.555059 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.557317 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:46.557325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:46.557335 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:46.557363 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:46.560270 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.560972 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.560979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.560982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.561111 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.561465 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.561471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.561474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.561792 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:46.562056 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.562332 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.562545 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:23:46.566713 14274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:23:46.573755 14274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:23:46.594352 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.584783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.0.97.1:42713)
0929 11:23:46.585830 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.585843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.586259 (+ 416us) server_negotiation.cc:407] Connection header received
0929 11:23:46.586670 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.586680 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.586942 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.587146 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:46.588146 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.589326 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.591849 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.593271 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.593417 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.593657 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.593855 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.594011 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":516,"threads_started":1}
I20260929 11:23:46.594396 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.584558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39394)
0929 11:23:46.586045 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.586110 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.586411 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.587439 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.587446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.587474 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:46.587912 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.587924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.589447 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.589455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.591682 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.591696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.593458 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.593463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.593634 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.593682 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.593819 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.594011 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":23,"thread_start_us":1456,"threads_started":1}
I20260929 11:23:46.595644 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.585112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35525 (local address 127.0.0.1:47730)
0929 11:23:46.586309 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.586331 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.586441 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.587623 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.587630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.587655 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:46.588134 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.588146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.590380 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.590384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.592385 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.592395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.594811 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.594817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.594966 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.595010 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.595110 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.595241 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":90,"thread_start_us":1126,"threads_started":1}
I20260929 11:23:46.595659 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.585834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.0.97.3:35525)
0929 11:23:46.587026 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.587037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.587072 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:23:46.587166 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.587174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.587402 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.588327 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:46.588654 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.590236 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.592529 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.593716 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.593852 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.594938 (+ 1086us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.595168 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.595316 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":890,"threads_started":1}
I20260929 11:23:46.596040 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.586801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33379 (local address 127.0.0.1:60960)
0929 11:23:46.587268 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.587283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.587382 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.588410 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.588414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.588441 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:46.588832 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.588848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.590857 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.590863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.593016 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.593033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.595324 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.595330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.595491 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.595523 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.595614 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.595743 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":589,"threads_started":1}
I20260929 11:23:46.597018 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.587232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.0.97.2:33379)
0929 11:23:46.587837 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.587846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.587872 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:23:46.587968 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.587988 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.588215 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.589041 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:46.589252 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.590711 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.593180 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.595096 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.596299 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.596468 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.596560 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.596741 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":1032,"threads_started":1}
I20260929 11:23:46.598591 14398 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5098c7d97045f58106bd63f8e04244 (DEFAULT_TABLE table=test-table [id=08ba30392a034d6ab3f97aa0722e8223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:46.599308 14665 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5098c7d97045f58106bd63f8e04244 (DEFAULT_TABLE table=test-table [id=08ba30392a034d6ab3f97aa0722e8223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:46.600086 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5098c7d97045f58106bd63f8e04244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:46.599804 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5098c7d97045f58106bd63f8e04244 (DEFAULT_TABLE table=test-table [id=08ba30392a034d6ab3f97aa0722e8223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:46.600693 14665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5098c7d97045f58106bd63f8e04244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:46.601032 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5098c7d97045f58106bd63f8e04244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:46.612144 14748 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Bootstrap starting.
I20260929 11:23:46.614239 14749 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Bootstrap starting.
I20260929 11:23:46.615792 14750 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Bootstrap starting.
I20260929 11:23:46.615852 14748 tablet_bootstrap.cc:654] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:46.617270 14748 log.cc:824] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:46.617638 14749 tablet_bootstrap.cc:654] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:46.618852 14750 tablet_bootstrap.cc:654] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:46.619050 14749 log.cc:824] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:46.620100 14750 log.cc:824] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:46.621133 14748 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: No bootstrap required, opened a new log
I20260929 11:23:46.621397 14748 ts_tablet_manager.cc:1397] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260929 11:23:46.621951 14749 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: No bootstrap required, opened a new log
I20260929 11:23:46.622226 14749 ts_tablet_manager.cc:1397] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 11:23:46.622642 14750 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: No bootstrap required, opened a new log
I20260929 11:23:46.622867 14750 ts_tablet_manager.cc:1397] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260929 11:23:46.629791 14750 raft_consensus.cc:348] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.630151 14750 raft_consensus.cc:374] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:46.630275 14750 raft_consensus.cc:729] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836d7249ccc94daaa9432c1fc7b766bd, State: Initialized, Role: FOLLOWER
I20260929 11:23:46.630894 14750 consensus_queue.cc:260] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.631917 14730 heartbeater.cc:503] Master 127.0.97.62:38265 was elected leader, sending a full tablet report...
I20260929 11:23:46.631740 14748 raft_consensus.cc:348] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.632169 14748 raft_consensus.cc:374] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:46.632270 14748 raft_consensus.cc:729] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50dc3297ef492e9ef7662bba3a63fc, State: Initialized, Role: FOLLOWER
I20260929 11:23:46.632501 14749 raft_consensus.cc:348] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.632921 14749 raft_consensus.cc:374] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:46.633023 14749 raft_consensus.cc:729] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16564e3d08c94134af84fc187db0a520, State: Initialized, Role: FOLLOWER
I20260929 11:23:46.632906 14748 consensus_queue.cc:260] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.633579 14749 consensus_queue.cc:260] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.634006 14750 ts_tablet_manager.cc:1428] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260929 11:23:46.635416 14597 heartbeater.cc:503] Master 127.0.97.62:38265 was elected leader, sending a full tablet report...
I20260929 11:23:46.635965 14749 ts_tablet_manager.cc:1428] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260929 11:23:46.636394 14748 ts_tablet_manager.cc:1428] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 11:23:46.643052 14755 raft_consensus.cc:482] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:23:46.643283 14755 raft_consensus.cc:504] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.644778 14755 leader_election.cc:288] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713), 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525)
I20260929 11:23:46.652923 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.645148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38489 (local address 127.0.97.1:42713)
0929 11:23:46.645357 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.645366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.646020 (+ 654us) server_negotiation.cc:407] Connection header received
0929 11:23:46.646530 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.646533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.646590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.646683 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:46.647294 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.648057 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.650933 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.651946 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.652059 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.652130 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.652532 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.652690 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:23:46.653081 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.645234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35625 (local address 127.0.97.3:35525)
0929 11:23:46.645423 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.645431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.646114 (+ 683us) server_negotiation.cc:407] Connection header received
0929 11:23:46.646256 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.646262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.646330 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.646395 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:46.646926 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.647865 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.650939 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.651965 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.652085 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.652197 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.652719 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.652846 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:23:46.653113 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.645079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.97.2:38489)
0929 11:23:46.645921 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.645937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.646116 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.646705 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.646710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.646739 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:46.647162 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.647175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.648183 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.648193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.650763 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.650778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.652162 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.652168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.652370 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.652412 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.652502 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.652670 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":90,"thread_start_us":498,"threads_started":1}
I20260929 11:23:46.653159 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.645259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35525 (local address 127.0.97.2:35625)
0929 11:23:46.646016 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.646034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:46.646136 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:46.646420 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:46.646423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:46.646441 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:46.646732 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.646739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.647986 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.647994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.650763 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:46.650778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.652376 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.652381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:46.652544 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.652570 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:23:46.652677 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 11:23:46.652793 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":189,"thread_start_us":446,"threads_started":1}
I20260929 11:23:46.654062 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5098c7d97045f58106bd63f8e04244" candidate_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16564e3d08c94134af84fc187db0a520" is_pre_election: true
I20260929 11:23:46.654201 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5098c7d97045f58106bd63f8e04244" candidate_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836d7249ccc94daaa9432c1fc7b766bd" is_pre_election: true
I20260929 11:23:46.654592 14418 raft_consensus.cc:2454] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50dc3297ef492e9ef7662bba3a63fc in term 0.
I20260929 11:23:46.654677 14685 raft_consensus.cc:2454] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50dc3297ef492e9ef7662bba3a63fc in term 0.
I20260929 11:23:46.655336 14488 leader_election.cc:302] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50dc3297ef492e9ef7662bba3a63fc, 836d7249ccc94daaa9432c1fc7b766bd; no voters:
I20260929 11:23:46.655762 14755 raft_consensus.cc:2781] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:23:46.655905 14755 raft_consensus.cc:482] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:23:46.656023 14755 raft_consensus.cc:3037] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:46.658315 14755 raft_consensus.cc:504] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.658859 14755 leader_election.cc:288] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [CANDIDATE]: Term 1 election: Requested vote from peers 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713), 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525)
I20260929 11:23:46.659260 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5098c7d97045f58106bd63f8e04244" candidate_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16564e3d08c94134af84fc187db0a520"
I20260929 11:23:46.659276 14685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5098c7d97045f58106bd63f8e04244" candidate_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836d7249ccc94daaa9432c1fc7b766bd"
I20260929 11:23:46.659451 14418 raft_consensus.cc:3037] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:46.659456 14685 raft_consensus.cc:3037] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:46.661662 14418 raft_consensus.cc:2454] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50dc3297ef492e9ef7662bba3a63fc in term 1.
I20260929 11:23:46.661691 14685 raft_consensus.cc:2454] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50dc3297ef492e9ef7662bba3a63fc in term 1.
I20260929 11:23:46.662098 14485 leader_election.cc:302] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16564e3d08c94134af84fc187db0a520, 2d50dc3297ef492e9ef7662bba3a63fc; no voters:
I20260929 11:23:46.662395 14755 raft_consensus.cc:2781] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:46.662837 14755 raft_consensus.cc:686] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 LEADER]: Becoming Leader. State: Replica: 2d50dc3297ef492e9ef7662bba3a63fc, State: Running, Role: LEADER
I20260929 11:23:46.663349 14755 consensus_queue.cc:237] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:46.667810 14272 catalog_manager.cc:5495] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } health_report { overall_health: HEALTHY } } }
I20260929 11:23:46.698415 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.693348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.97.2:33379)
0929 11:23:46.693556 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.693560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.693649 (+ 89us) server_negotiation.cc:407] Connection header received
0929 11:23:46.693809 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.693813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.693889 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.694024 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:23:46.694337 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.695026 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.696424 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.696920 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.697026 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.697490 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:23:46.697973 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:23:46.698078 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.698139 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.698200 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:23:46.706348 14685 raft_consensus.cc:1261] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Refusing update from remote peer 2d50dc3297ef492e9ef7662bba3a63fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:46.706348 14418 raft_consensus.cc:1261] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Refusing update from remote peer 2d50dc3297ef492e9ef7662bba3a63fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:46.707361 14759 consensus_queue.cc:1035] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:46.707732 14755 consensus_queue.cc:1035] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:46.722890 14763 mvcc.cc:205] Tried to move back new op lower bound from 7334629485378932736 to 7334629485216468992. Current Snapshot: MvccSnapshot[applied={T|T < 7334629485378932736}]
I20260929 11:23:46.731319 14272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:23:46.731573 14272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:23:46.734961 14272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Sending DeleteTablet for 3 replicas of tablet 0d5098c7d97045f58106bd63f8e04244
I20260929 11:23:46.736074 14398 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5098c7d97045f58106bd63f8e04244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39394
I20260929 11:23:46.736287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14335
I20260929 11:23:46.736449 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5098c7d97045f58106bd63f8e04244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60960
I20260929 11:23:46.736825 14665 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5098c7d97045f58106bd63f8e04244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47730
I20260929 11:23:46.760972 14774 tablet_replica.cc:330] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: stopping tablet replica
I20260929 11:23:46.761668 14774 raft_consensus.cc:2229] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:23:46.762332 14774 raft_consensus.cc:2258] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 11:23:46 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:23:46.763320 14772 tablet_replica.cc:330] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: stopping tablet replica
I20260929 11:23:46.764014 14772 raft_consensus.cc:2229] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:23:46.764727 14772 raft_consensus.cc:2258] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:46.767416 14774 ts_tablet_manager.cc:1905] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:46.770896 14772 ts_tablet_manager.cc:1905] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:46.772428 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.737438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.97.1:42713)
0929 11:23:46.737662 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.737671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.749651 (+ 11980us) server_negotiation.cc:407] Connection header received
0929 11:23:46.750205 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.750215 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.750343 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.750427 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:46.752799 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.754238 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.756903 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.757731 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.757846 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.764118 (+ 6272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:46.764136 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:46.764156 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:46.764199 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:46.768054 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.769033 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.769041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.769045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.769207 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.769752 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.769757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.769760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.770247 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:46.770525 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.771127 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.771353 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:23:46.773111 14458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:46.779896 14774 ts_tablet_manager.cc:1918] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:46.780757 14774 log.cc:1196] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/0d5098c7d97045f58106bd63f8e04244
I20260929 11:23:46.780761 14773 tablet_replica.cc:330] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: stopping tablet replica
I20260929 11:23:46.781291 14774 ts_tablet_manager.cc:1939] T 0d5098c7d97045f58106bd63f8e04244 P 836d7249ccc94daaa9432c1fc7b766bd: Deleting consensus metadata
I20260929 11:23:46.781548 14773 raft_consensus.cc:2229] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:23:46.782403 14773 raft_consensus.cc:2258] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:46.783118 14261 catalog_manager.cc:4841] TS 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525): tablet 0d5098c7d97045f58106bd63f8e04244 (table test-table [id=08ba30392a034d6ab3f97aa0722e8223]) successfully deleted
I20260929 11:23:46.785388 14773 ts_tablet_manager.cc:1905] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:46.791069 14773 ts_tablet_manager.cc:1918] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:46.791239 14773 log.cc:1196] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/0d5098c7d97045f58106bd63f8e04244
I20260929 11:23:46.791700 14773 ts_tablet_manager.cc:1939] T 0d5098c7d97045f58106bd63f8e04244 P 2d50dc3297ef492e9ef7662bba3a63fc: Deleting consensus metadata
I20260929 11:23:46.793095 14261 catalog_manager.cc:4841] TS 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2:33379): tablet 0d5098c7d97045f58106bd63f8e04244 (table test-table [id=08ba30392a034d6ab3f97aa0722e8223]) successfully deleted
I20260929 11:23:46.889415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14335
W20260929 11:23:46.907684 14258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42713 (error 108)
I20260929 11:23:46.907829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14469
I20260929 11:23:46.909638 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.909248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39414)
0929 11:23:46.909411 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.909510 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 11:23:46.909933 14258 catalog_manager.cc:4584] TS 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d5098c7d97045f58106bd63f8e04244: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Sep 29 11:23:46 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:23:46.927474 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.908690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.0.97.2:33379)
0929 11:23:46.908876 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:46.908883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:46.909133 (+ 250us) server_negotiation.cc:407] Connection header received
0929 11:23:46.909208 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:46.909214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:46.909289 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:46.909389 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:46.911136 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.911916 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.913915 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:46.914558 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:46.914694 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:46.920121 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:46.920141 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:46.920167 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:46.920211 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:46.924242 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.925042 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.925051 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.925054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.925221 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:46.925618 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:46.925624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:46.925627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:46.926095 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:46.926409 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:46.926691 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 11:23:46.926919 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:23:46.928261 14592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:46.963656 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.963198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39424)
0929 11:23:46.963406 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.963501 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 11:23:46.964792 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:46.964318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39432)
0929 11:23:46.964459 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:46.964546 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 11:23:47.025178 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:47.024720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39440)
0929 11:23:47.024912 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:47.024996 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 11:23:47.026182 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:47.025850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39446)
0929 11:23:47.026008 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:47.026080 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 11:23:47.042142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14469
I20260929 11:23:47.060515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14602
Sep 29 11:23:47 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:23:47.075775 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:47.061299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.0.97.3:35525)
0929 11:23:47.061474 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:47.061480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:47.061586 (+ 106us) server_negotiation.cc:407] Connection header received
0929 11:23:47.061824 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:47.061828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:47.061895 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:47.062010 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:47.063378 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:47.064130 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:47.065464 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:47.065866 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:47.065998 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:47.070290 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:47.070304 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:47.070320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:47.070350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:47.073061 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:47.073728 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:47.073736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:47.073738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:47.073869 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:47.074266 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:47.074270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:47.074272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:47.074617 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:47.074909 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:47.075233 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 11:23:47.075455 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:23:47.076418 14725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:47.106302 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:47.105875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39460)
0929 11:23:47.106063 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:47.106157 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 11:23:47.107329 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:47.106959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39476)
0929 11:23:47.107154 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:47.107227 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 11:23:47.164669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14602
I20260929 11:23:47.181550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14242
I20260929 11:23:47.182615 14304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:47.292045 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:47.291235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39480)
0929 11:23:47.291513 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:47.291636 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 11:23:47.292092 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14242
I20260929 11:23:47.312435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38265
--webserver_interface=127.0.97.62
--webserver_port=36275
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:47.630209 14781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:47.630527 14781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:47.630633 14781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:47.638281 14781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:47.638398 14781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:47.638449 14781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:47.638541 14781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:47.638629 14781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:47.649338 14781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:47.650784 14781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:47.652244 14781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:47.674412 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:47.674350 14788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:47.674448 14790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:47.675333 14781 server_base.cc:1034] running on GCE node
I20260929 11:23:47.676165 14781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:47.677772 14781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:47.679014 14781 hybrid_clock.cc:648] HybridClock initialized: now 1790681027678962 us; error 60 us; skew 500 ppm
I20260929 11:23:47.679417 14781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:47 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:47.682965 14781 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:47.684747 14781 webserver.cc:460] Webserver started at http://127.0.97.62:36275/ using document root <none> and password file <none>
I20260929 11:23:47.685494 14781 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:47.685665 14781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:47.691490 14781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 11:23:47.694337 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:47.695571 14781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:23:47.695766 14781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "f040e582a6a5490f9e25fd428ba022e8"
format_stamp: "Formatted at 2026-09-29 11:23:44 on dist-test-slave-srst"
server_key: "904d6887dffeb784350f5c95c22ab60d"
server_key_iv: "f158138d1e16592449b2ecd1cfe1e44e"
server_key_version: "encryptionkey@0"
I20260929 11:23:47.696943 14781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:47.715252 14781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:47.719152 14781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:47.719455 14781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:47.737661 14781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38265
I20260929 11:23:47.737681 14848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38265 every 8 connection(s)
I20260929 11:23:47.739107 14781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:47.743595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14781
I20260929 11:23:47.744333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42713
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38265
--webserver_port=44365
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:47.745821 14849 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:23:47.750301 14849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap starting.
I20260929 11:23:47.770305 14849 log.cc:824] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:47.781896 14849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:23:47.782434 14849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap complete.
I20260929 11:23:47.791268 14849 raft_consensus.cc:348] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:47.792045 14849 raft_consensus.cc:729] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f040e582a6a5490f9e25fd428ba022e8, State: Initialized, Role: FOLLOWER
I20260929 11:23:47.792737 14849 consensus_queue.cc:260] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:47.792999 14849 raft_consensus.cc:388] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:23:47.793131 14849 raft_consensus.cc:482] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:23:47.793313 14849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:23:47.796995 14849 raft_consensus.cc:504] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:47.797506 14849 leader_election.cc:302] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f040e582a6a5490f9e25fd428ba022e8; no voters:
I20260929 11:23:47.798147 14849 leader_election.cc:288] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:23:47.798280 14853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:23:47.799444 14853 raft_consensus.cc:686] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 LEADER]: Becoming Leader. State: Replica: f040e582a6a5490f9e25fd428ba022e8, State: Running, Role: LEADER
I20260929 11:23:47.800097 14853 consensus_queue.cc:237] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:47.800540 14849 sys_catalog.cc:564] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:47.802773 14855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f040e582a6a5490f9e25fd428ba022e8. Latest consensus state: current_term: 2 leader_uuid: "f040e582a6a5490f9e25fd428ba022e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } } }
I20260929 11:23:47.803121 14855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:47.802858 14854 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f040e582a6a5490f9e25fd428ba022e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } } }
I20260929 11:23:47.803248 14854 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:47.817670 14859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:47.825565 14859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:47.827081 14859 catalog_manager.cc:1259] Loaded cluster ID: 6336938a83794aec927b3d87caa0cda7
I20260929 11:23:47.827252 14859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:23:47.830490 14859 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 11:23:47.847805 14859 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 11:23:47.857012 14859 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:23:47.858932 14859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Loaded TSK: 0
I20260929 11:23:47.860736 14859 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:23:48.090915 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:48.091197 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:48.091349 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:48.098616 14851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:48.098716 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:48.098875 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:23:48.108767 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44365
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:48.110038 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:48.111411 14851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:48.125304 14876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:48.125326 14877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:48.129025 14879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:48.130681 14851 server_base.cc:1034] running on GCE node
I20260929 11:23:48.131484 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:48.132763 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:48.134016 14851 hybrid_clock.cc:648] HybridClock initialized: now 1790681028133963 us; error 40 us; skew 500 ppm
I20260929 11:23:48.134405 14851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:48 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:48.138145 14851 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:23:48.139715 14851 webserver.cc:460] Webserver started at http://127.0.97.1:44365/ using document root <none> and password file <none>
I20260929 11:23:48.140405 14851 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:48.140551 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:48.145790 14851 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:23:48.148905 14886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:48.150280 14851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:23:48.150442 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "16564e3d08c94134af84fc187db0a520"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "9e59de2633192efce0db75aae2b7934f"
server_key_iv: "6ba4277e72d7b670520592508d4b9dc3"
server_key_version: "encryptionkey@0"
I20260929 11:23:48.151396 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:48.177376 14851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:48.181417 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:48.181715 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:48.182706 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:48.184468 14893 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 11:23:48.190539 14851 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 11:23:48.190687 14851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260929 11:23:48.190834 14851 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 11:23:48.194010 14851 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 11:23:48.194110 14851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260929 11:23:48.194461 14893 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Bootstrap starting.
I20260929 11:23:48.203286 14893 log.cc:824] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:48.217491 14893 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:23:48.218082 14893 tablet_bootstrap.cc:492] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Bootstrap complete.
I20260929 11:23:48.218518 14893 ts_tablet_manager.cc:1397] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.001s
I20260929 11:23:48.227613 14893 raft_consensus.cc:348] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:48.229259 14893 raft_consensus.cc:729] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16564e3d08c94134af84fc187db0a520, State: Initialized, Role: FOLLOWER
I20260929 11:23:48.229900 14893 consensus_queue.cc:260] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260929 11:23:48.231468 14893 ts_tablet_manager.cc:1428] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 11:23:48.240793 14851 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42713
I20260929 11:23:48.240830 15001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42713 every 8 connection(s)
I20260929 11:23:48.241964 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:23:48.248495 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 14851
I20260929 11:23:48.249317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33379
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38265
--webserver_port=38899
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:23:48 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:48.266258 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:48.244767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52369 (local address 127.0.97.62:38265)
0929 11:23:48.245412 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:48.245427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:48.245737 (+ 310us) server_negotiation.cc:407] Connection header received
0929 11:23:48.246566 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:48.246579 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:48.246962 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:48.247303 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:48.249451 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.250826 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.253409 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.254069 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.254227 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:48.258828 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:48.258836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:48.258851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:48.258892 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:48.261791 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:48.262804 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:48.262811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:48.262816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:48.262963 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:48.263428 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:48.263431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:48.263433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:48.263871 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:48.264007 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:48.264992 (+ 985us) server_negotiation.cc:299] Negotiation successful
0929 11:23:48.265308 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":265,"threads_started":1}
I20260929 11:23:48.266749 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:48.244585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:52369)
0929 11:23:48.245584 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:48.245617 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:48.246067 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:48.247430 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:48.247442 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:48.248312 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:48.249154 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:48.249197 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.251019 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.251027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:48.253246 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:48.253266 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.254299 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.254308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:48.254653 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:48.255437 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:48.255471 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:48.258554 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:48.261941 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:48.261952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:48.261970 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:48.262668 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:48.263076 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:48.263083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:48.263089 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:48.263319 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:48.264028 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:48.264039 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:48.264454 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 11:23:48.264811 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 11:23:48.265382 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":442,"threads_started":1}
I20260929 11:23:48.269647 15002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38265
I20260929 11:23:48.269943 15002 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:48.270637 15002 heartbeater.cc:511] Master 127.0.97.62:38265 requested a full tablet report, sending...
I20260929 11:23:48.273706 14814 ts_manager.cc:194] Registered new tserver with Master: 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713)
I20260929 11:23:48.278172 14814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52369
I20260929 11:23:48.289917 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:48.277035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.0.1:39496)
0929 11:23:48.283824 (+ 6789us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:48.283859 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:48.284060 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:48.284738 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:48.284745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:48.284778 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:48.285113 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:48.285123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.286387 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.286394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:48.287744 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:48.287756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.288992 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.288997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:48.289180 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:48.289243 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 11:23:48.289368 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 11:23:48.289537 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6593,"thread_start_us":6499,"threads_started":1}
I20260929 11:23:48.289916 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:48.277209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.0.97.1:42713)
0929 11:23:48.283612 (+ 6403us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:48.283621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:48.283968 (+ 347us) server_negotiation.cc:407] Connection header received
0929 11:23:48.284293 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:48.284299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:48.284519 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:48.284738 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:48.285287 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.286266 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.287901 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.288828 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.288967 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:48.289168 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:48.289409 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 11:23:48.289549 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6218,"thread_start_us":6062,"threads_started":1}
I20260929 11:23:48.291432 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5098c7d97045f58106bd63f8e04244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39496
I20260929 11:23:48.293841 15009 tablet_replica.cc:330] stopping tablet replica
I20260929 11:23:48.294183 15009 raft_consensus.cc:2229] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:23:48.294606 15009 raft_consensus.cc:2258] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:48.295737 15009 ts_tablet_manager.cc:1905] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:48.302589 15009 ts_tablet_manager.cc:1918] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:48.302737 15009 log.cc:1196] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/0d5098c7d97045f58106bd63f8e04244
I20260929 11:23:48.303202 15009 ts_tablet_manager.cc:1939] T 0d5098c7d97045f58106bd63f8e04244 P 16564e3d08c94134af84fc187db0a520: Deleting consensus metadata
I20260929 11:23:48.305311 14798 catalog_manager.cc:4841] TS 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713): tablet 0d5098c7d97045f58106bd63f8e04244 (table test-table [id=08ba30392a034d6ab3f97aa0722e8223]) successfully deleted
W20260929 11:23:48.582124 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:48.582409 15006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:48.582500 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:48.589398 15006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:48.589504 15006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:48.589612 15006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:23:48.599171 15006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38899
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:48.600682 15006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:48.602105 15006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:48.615042 15015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:48.615193 15014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:48.615180 15017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:48.615821 15006 server_base.cc:1034] running on GCE node
I20260929 11:23:48.616495 15006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:48.617412 15006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:48.618607 15006 hybrid_clock.cc:648] HybridClock initialized: now 1790681028618564 us; error 41 us; skew 500 ppm
I20260929 11:23:48.618925 15006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:48 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:48.621877 15006 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:23:48.623335 15006 webserver.cc:460] Webserver started at http://127.0.97.2:38899/ using document root <none> and password file <none>
I20260929 11:23:48.624020 15006 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:48.624194 15006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:48.629351 15006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:23:48.631852 15024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:48.633139 15006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:23:48.633315 15006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "2d50dc3297ef492e9ef7662bba3a63fc"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "688bac4cd5633f0302e07bd4afa12e8a"
server_key_iv: "e34b60573c0c7bd53eec618c6730fd8a"
server_key_version: "encryptionkey@0"
I20260929 11:23:48.634231 15006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:48.652118 15006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:48.656075 15006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:48.656350 15006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:48.657322 15006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:48.658577 15006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:48.658700 15006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:48.658864 15006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:48.658954 15006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:48.691383 15006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33379
I20260929 11:23:48.691470 15136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33379 every 8 connection(s)
I20260929 11:23:48.692895 15006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:23:48.700277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15006
I20260929 11:23:48.700965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35525
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38265
--webserver_port=33617
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:23:48 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:48.712378 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:48.695442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34459 (local address 127.0.97.62:38265)
0929 11:23:48.695624 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:48.695629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:48.696764 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 11:23:48.698562 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:48.698567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:48.698670 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:48.698776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:48.700257 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.701086 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.702650 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.703073 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.703225 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:48.706562 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:48.706567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:48.706570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:48.706601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:48.709389 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:48.710097 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:48.710103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:48.710107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:48.710252 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:48.710682 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:48.710688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:48.710691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:48.710964 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:48.711120 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:48.711793 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 11:23:48.712014 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:23:48.713330 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:48.695888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:34459)
0929 11:23:48.696560 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:48.696653 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:48.698309 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:48.699025 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:48.699034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:48.699545 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:48.700090 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:48.700113 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.701252 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.701259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:48.702508 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:48.702520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:48.703306 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:48.703315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:48.703550 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:48.704219 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:48.704242 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:48.706349 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:48.709541 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:48.709550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:48.709566 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:48.709967 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:48.710360 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:48.710365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:48.710369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:48.710567 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:48.711090 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:48.711097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:48.711425 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 11:23:48.711706 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 11:23:48.712234 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":263,"threads_started":1}
I20260929 11:23:48.714799 15137 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38265
I20260929 11:23:48.715132 15137 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:48.716010 15137 heartbeater.cc:511] Master 127.0.97.62:38265 requested a full tablet report, sending...
I20260929 11:23:48.717667 14814 ts_manager.cc:194] Registered new tserver with Master: 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2:33379)
I20260929 11:23:48.718978 14814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34459
W20260929 11:23:49.018673 15140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:49.018937 15140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:49.019124 15140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:49.027230 15140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:49.027365 15140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:49.027544 15140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:23:49.037283 15140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33617
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:49.038633 15140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:49.040004 15140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:49.052991 15148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:49.053082 15146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:49.053144 15145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:49.054055 15140 server_base.cc:1034] running on GCE node
I20260929 11:23:49.054637 15140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:49.055651 15140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:49.056876 15140 hybrid_clock.cc:648] HybridClock initialized: now 1790681029056825 us; error 60 us; skew 500 ppm
I20260929 11:23:49.057184 15140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:49 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:49.060268 15140 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:23:49.061743 15140 webserver.cc:460] Webserver started at http://127.0.97.3:33617/ using document root <none> and password file <none>
I20260929 11:23:49.062424 15140 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:49.062577 15140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:49.067656 15140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 11:23:49.070238 15155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:49.071542 15140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:23:49.071707 15140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "836d7249ccc94daaa9432c1fc7b766bd"
format_stamp: "Formatted at 2026-09-29 11:23:46 on dist-test-slave-srst"
server_key: "6857097f065968a883e65011dcdd3ae5"
server_key_iv: "5497cc979807cc4bdca528a75a6d5546"
server_key_version: "encryptionkey@0"
I20260929 11:23:49.072714 15140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:49.094172 15140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:49.098676 15140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:49.098953 15140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:49.099972 15140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:49.101459 15140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:49.101594 15140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:49.101734 15140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:49.101814 15140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:49.134904 15140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35525
I20260929 11:23:49.134956 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35525 every 8 connection(s)
I20260929 11:23:49.136371 15140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:23:49.143090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15140
Sep 29 11:23:49 dist-test-slave-srst krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:23:49.157966 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.139131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34357 (local address 127.0.97.62:38265)
0929 11:23:49.139333 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.139340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.140497 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 11:23:49.142239 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.142243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.142305 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.142395 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:49.143881 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.144895 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.147218 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.147765 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.147904 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.151368 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:49.151378 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:49.151381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:49.151411 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:49.154638 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.155489 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.155495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.155499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.155625 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.156047 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.156053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.156056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.156395 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:49.156560 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.157381 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.157586 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":67}
I20260929 11:23:49.158473 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.143915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.97.62:38265)
0929 11:23:49.145022 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.145026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.145039 (+ 13us) server_negotiation.cc:407] Connection header received
0929 11:23:49.145075 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.145078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.145126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.145201 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:49.147042 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.147911 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.149535 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.149926 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.150057 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.153015 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:49.153019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:49.153022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:49.153059 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:49.155746 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.156407 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.156411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.156417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.156533 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.156958 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.156971 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.156974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.157206 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:49.157463 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.157803 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.158016 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1029,"thread_start_us":895,"threads_started":1}
I20260929 11:23:49.159229 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.139669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:34357)
0929 11:23:49.140344 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.140383 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.142002 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.142596 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.142603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.143081 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:23:49.143668 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.143705 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.145050 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.145062 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.147088 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.147100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.147992 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.147999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.148247 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.148992 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:23:49.149017 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:23:49.151163 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:23:49.154819 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:49.154827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:49.154843 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:49.155364 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:49.155749 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:23:49.155753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:23:49.155757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:23:49.155937 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:23:49.156531 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:23:49.156538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:23:49.157015 (+ 477us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.157274 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.157780 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":263,"threads_started":1}
I20260929 11:23:49.160787 15268 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38265
I20260929 11:23:49.161092 15268 heartbeater.cc:464] Registering TS with master...
I20260929 11:23:49.161774 15268 heartbeater.cc:511] Master 127.0.97.62:38265 requested a full tablet report, sending...
I20260929 11:23:49.163067 14814 ts_manager.cc:194] Registered new tserver with Master: 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525)
I20260929 11:23:49.163913 14814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34357
I20260929 11:23:49.173239 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:23:49.179584 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.174073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.97.62:38265)
0929 11:23:49.174260 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.174282 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.174716 (+ 434us) server_negotiation.cc:407] Connection header received
0929 11:23:49.174886 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.174889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.175013 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.175142 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:23:49.175479 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.176180 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.177718 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.178115 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.178206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.178643 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:23:49.179141 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:23:49.179244 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.179335 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.179395 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:23:49.180456 14814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:23:49.187547 14814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:23:49.199436 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf31c4c16b441f9a284d775eae70732 (DEFAULT_TABLE table=test-table [id=fbc08931c5da4d6d9b1427af33ac4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:49.200441 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf31c4c16b441f9a284d775eae70732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:49.208926 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.197863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33379 (local address 127.0.0.1:32768)
0929 11:23:49.198576 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.198600 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.198745 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.199870 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.199876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.199899 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:49.200245 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.200255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.202650 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.202655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.205425 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.205450 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.207771 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.207780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.208042 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.208100 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.208184 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.208374 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":376,"threads_started":1}
I20260929 11:23:49.210335 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.198113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.97.2:33379)
0929 11:23:49.199256 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.199265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.199299 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:23:49.199407 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.199417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.199672 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.200411 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:49.200700 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.202509 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.205700 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.207490 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.209494 (+ 2004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.209752 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.209844 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.210019 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":862,"threads_started":1}
I20260929 11:23:49.211622 15282 tablet_bootstrap.cc:492] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Bootstrap starting.
I20260929 11:23:49.213460 15282 tablet_bootstrap.cc:654] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:49.214541 15072 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf31c4c16b441f9a284d775eae70732 (DEFAULT_TABLE table=test-table [id=fbc08931c5da4d6d9b1427af33ac4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:49.215653 15282 tablet_bootstrap.cc:492] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: No bootstrap required, opened a new log
I20260929 11:23:49.215802 15282 ts_tablet_manager.cc:1397] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 11:23:49.215946 15072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf31c4c16b441f9a284d775eae70732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:49.216694 15282 raft_consensus.cc:348] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.216943 15282 raft_consensus.cc:374] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:49.217056 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.198561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35525 (local address 127.0.0.1:47736)
0929 11:23:49.209159 (+ 10598us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.209176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.209288 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.210730 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.210734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.210758 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:49.210931 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.210940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.212631 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.212636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.213853 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.213864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.216276 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.216281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.216461 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.216501 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.216622 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.216765 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10516,"thread_start_us":11401,"threads_started":1}
I20260929 11:23:49.217057 15282 raft_consensus.cc:729] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16564e3d08c94134af84fc187db0a520, State: Initialized, Role: FOLLOWER
I20260929 11:23:49.217341 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.200943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.0.97.3:35525)
0929 11:23:49.209906 (+ 8963us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.209915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.209950 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:23:49.210052 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.210060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.210361 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.210579 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:49.211108 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.212469 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.213997 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.216121 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.216243 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.216453 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.216678 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.216848 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8778,"thread_start_us":8625,"threads_started":1}
I20260929 11:23:49.217339 15282 consensus_queue.cc:260] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.218189 15002 heartbeater.cc:503] Master 127.0.97.62:38265 was elected leader, sending a full tablet report...
I20260929 11:23:49.218811 15282 ts_tablet_manager.cc:1428] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:23:49.220878 15203 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf31c4c16b441f9a284d775eae70732 (DEFAULT_TABLE table=test-table [id=fbc08931c5da4d6d9b1427af33ac4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:23:49.222205 15203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf31c4c16b441f9a284d775eae70732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:23:49.230975 15285 tablet_bootstrap.cc:492] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Bootstrap starting.
I20260929 11:23:49.233752 15285 tablet_bootstrap.cc:654] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:49.234736 15285 log.cc:824] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:49.235466 15286 tablet_bootstrap.cc:492] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Bootstrap starting.
I20260929 11:23:49.237752 15285 tablet_bootstrap.cc:492] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: No bootstrap required, opened a new log
I20260929 11:23:49.238022 15285 ts_tablet_manager.cc:1397] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 11:23:49.238656 15286 tablet_bootstrap.cc:654] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Neither blocks nor log segments found. Creating new log.
I20260929 11:23:49.239774 15286 log.cc:824] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:49.242739 15286 tablet_bootstrap.cc:492] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: No bootstrap required, opened a new log
I20260929 11:23:49.243008 15286 ts_tablet_manager.cc:1397] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
W20260929 11:23:49.244747 15003 tablet.cc:2369] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:23:49.248059 15285 raft_consensus.cc:348] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.248466 15285 raft_consensus.cc:374] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:49.248551 15285 raft_consensus.cc:729] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50dc3297ef492e9ef7662bba3a63fc, State: Initialized, Role: FOLLOWER
I20260929 11:23:49.249127 15285 consensus_queue.cc:260] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.250072 15137 heartbeater.cc:503] Master 127.0.97.62:38265 was elected leader, sending a full tablet report...
I20260929 11:23:49.250617 15285 ts_tablet_manager.cc:1428] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 11:23:49.252916 15286 raft_consensus.cc:348] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.253239 15286 raft_consensus.cc:374] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:23:49.253361 15286 raft_consensus.cc:729] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836d7249ccc94daaa9432c1fc7b766bd, State: Initialized, Role: FOLLOWER
I20260929 11:23:49.253927 15286 consensus_queue.cc:260] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.255023 15268 heartbeater.cc:503] Master 127.0.97.62:38265 was elected leader, sending a full tablet report...
I20260929 11:23:49.255610 15286 ts_tablet_manager.cc:1428] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260929 11:23:49.388192 15269 tablet.cc:2369] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:23:49.444890 15138 tablet.cc:2369] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:23:49.506944 15284 raft_consensus.cc:482] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:23:49.507272 15284 raft_consensus.cc:504] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.508739 15284 leader_election.cc:288] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2:33379), 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525)
I20260929 11:23:49.511945 15289 raft_consensus.cc:482] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:23:49.512253 15289 raft_consensus.cc:504] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.514281 15289 leader_election.cc:288] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525), 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713)
I20260929 11:23:49.517071 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.510026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40069 (local address 127.0.97.3:35525)
0929 11:23:49.510269 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.510275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.510641 (+ 366us) server_negotiation.cc:407] Connection header received
0929 11:23:49.510781 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.510786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.510867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.510955 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:49.511620 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.512515 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.514400 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.515718 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.515845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.515971 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.516617 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.516776 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:23:49.517752 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.509868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35525 (local address 127.0.97.1:40069)
0929 11:23:49.510547 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.510565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.510664 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.510983 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.510987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.511009 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:49.511400 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.511412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.512709 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.512715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.514205 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.514222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.516178 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.516187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.516419 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.516453 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.517258 (+ 805us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.517426 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":335,"threads_started":1}
I20260929 11:23:49.518807 15223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf31c4c16b441f9a284d775eae70732" candidate_uuid: "16564e3d08c94134af84fc187db0a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836d7249ccc94daaa9432c1fc7b766bd" is_pre_election: true
I20260929 11:23:49.519202 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.509096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33379 (local address 127.0.97.1:60777)
0929 11:23:49.509722 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.509743 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.509881 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.510401 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.510405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.510430 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:49.510819 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.510832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.512259 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.512264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.514416 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.514431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.518441 (+ 4010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.518446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.518651 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.518680 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.518768 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.518890 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":369,"threads_started":1}
I20260929 11:23:49.519351 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.509224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60777 (local address 127.0.97.2:33379)
0929 11:23:49.509412 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.509417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.509830 (+ 413us) server_negotiation.cc:407] Connection header received
0929 11:23:49.510036 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.510041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.510122 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.510235 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:49.511001 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.512079 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.514612 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.518270 (+ 3658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.518374 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.518488 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.519008 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.519130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:23:49.519382 15223 raft_consensus.cc:2454] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16564e3d08c94134af84fc187db0a520 in term 0.
I20260929 11:23:49.520117 14890 leader_election.cc:302] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16564e3d08c94134af84fc187db0a520, 836d7249ccc94daaa9432c1fc7b766bd; no voters:
I20260929 11:23:49.520548 15284 raft_consensus.cc:2781] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:23:49.520370 15092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf31c4c16b441f9a284d775eae70732" candidate_uuid: "16564e3d08c94134af84fc187db0a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" is_pre_election: true
I20260929 11:23:49.520730 15284 raft_consensus.cc:482] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:23:49.520825 15284 raft_consensus.cc:3037] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:49.520925 15092 raft_consensus.cc:2454] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16564e3d08c94134af84fc187db0a520 in term 0.
I20260929 11:23:49.524001 15284 raft_consensus.cc:504] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.524178 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.515218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50289 (local address 127.0.97.3:35525)
0929 11:23:49.516286 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.516294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.517174 (+ 880us) server_negotiation.cc:407] Connection header received
0929 11:23:49.517344 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.517349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.517419 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.517507 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:49.518398 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.519624 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.522208 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.523212 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.523351 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.523464 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.523728 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.523865 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":653,"threads_started":1}
I20260929 11:23:49.524364 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.514623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35525 (local address 127.0.97.2:50289)
0929 11:23:49.517000 (+ 2377us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.517019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.517198 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.517722 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.517727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.517752 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:49.518201 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.518212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.519790 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.519798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.522055 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.522070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.523427 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.523433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.523585 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.523608 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.523709 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.523848 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"thread_start_us":4222,"threads_started":1}
I20260929 11:23:49.524742 15284 leader_election.cc:288] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [CANDIDATE]: Term 1 election: Requested vote from peers 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2:33379), 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525)
I20260929 11:23:49.525106 15223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf31c4c16b441f9a284d775eae70732" candidate_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836d7249ccc94daaa9432c1fc7b766bd" is_pre_election: true
I20260929 11:23:49.525216 15092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf31c4c16b441f9a284d775eae70732" candidate_uuid: "16564e3d08c94134af84fc187db0a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50dc3297ef492e9ef7662bba3a63fc"
I20260929 11:23:49.525331 15223 raft_consensus.cc:2454] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50dc3297ef492e9ef7662bba3a63fc in term 0.
I20260929 11:23:49.525264 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf31c4c16b441f9a284d775eae70732" candidate_uuid: "16564e3d08c94134af84fc187db0a520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836d7249ccc94daaa9432c1fc7b766bd"
I20260929 11:23:49.525384 15092 raft_consensus.cc:3037] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:49.525537 15222 raft_consensus.cc:3037] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:23:49.525827 15028 leader_election.cc:302] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50dc3297ef492e9ef7662bba3a63fc, 836d7249ccc94daaa9432c1fc7b766bd; no voters:
I20260929 11:23:49.527710 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.515093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36773 (local address 127.0.97.1:42713)
0929 11:23:49.516753 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.516756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.518026 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 11:23:49.518083 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.518087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.518154 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.519718 (+ 1564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:23:49.521053 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.522729 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.525090 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.526324 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.526447 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.526564 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.527240 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.527385 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":1429,"threads_started":1}
I20260929 11:23:49.527896 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.514748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42713 (local address 127.0.97.2:36773)
0929 11:23:49.517736 (+ 2988us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:49.517756 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:23:49.517913 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:23:49.519985 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:23:49.519992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:23:49.520031 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:23:49.520605 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.520619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.522879 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.522886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.524910 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:23:49.524923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.526826 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.526832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:23:49.527012 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.527047 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:23:49.527149 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:23:49.527288 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2858,"thread_start_us":3931,"threads_started":1}
I20260929 11:23:49.528533 15222 raft_consensus.cc:2454] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16564e3d08c94134af84fc187db0a520 in term 1.
I20260929 11:23:49.528695 15092 raft_consensus.cc:2454] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16564e3d08c94134af84fc187db0a520 in term 1.
I20260929 11:23:49.528982 15289 raft_consensus.cc:2741] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 11:23:49.528770 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf31c4c16b441f9a284d775eae70732" candidate_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16564e3d08c94134af84fc187db0a520" is_pre_election: true
I20260929 11:23:49.528998 14890 leader_election.cc:302] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16564e3d08c94134af84fc187db0a520, 836d7249ccc94daaa9432c1fc7b766bd; no voters:
I20260929 11:23:49.529306 14956 raft_consensus.cc:2379] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d50dc3297ef492e9ef7662bba3a63fc in current term 1: Already voted for candidate 16564e3d08c94134af84fc187db0a520 in this term.
I20260929 11:23:49.529567 15284 raft_consensus.cc:2781] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:23:49.530118 15284 raft_consensus.cc:686] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 1 LEADER]: Becoming Leader. State: Replica: 16564e3d08c94134af84fc187db0a520, State: Running, Role: LEADER
I20260929 11:23:49.530531 15284 consensus_queue.cc:237] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } }
I20260929 11:23:49.534773 14814 catalog_manager.cc:5495] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16564e3d08c94134af84fc187db0a520 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "16564e3d08c94134af84fc187db0a520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16564e3d08c94134af84fc187db0a520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42713 } health_report { overall_health: HEALTHY } } }
I20260929 11:23:49.574409 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.569290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.0.97.1:42713)
0929 11:23:49.569471 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.569475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.569492 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:23:49.569647 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.569653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.569780 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.569918 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:23:49.570225 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.570924 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.572270 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.572734 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.572865 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.573287 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:23:49.573829 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:23:49.574000 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.574051 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.574130 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:23:49.580332 15092 raft_consensus.cc:1261] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Refusing update from remote peer 16564e3d08c94134af84fc187db0a520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:49.580376 15222 raft_consensus.cc:1261] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Refusing update from remote peer 16564e3d08c94134af84fc187db0a520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:23:49.581209 15297 consensus_queue.cc:1035] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50dc3297ef492e9ef7662bba3a63fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:49.581585 15284 consensus_queue.cc:1035] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836d7249ccc94daaa9432c1fc7b766bd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:23:49.592242 15300 mvcc.cc:205] Tried to move back new op lower bound from 7334629497152155648 to 7334629496961196032. Current Snapshot: MvccSnapshot[applied={T|T < 7334629497152155648}]
I20260929 11:23:49.596742 15301 mvcc.cc:205] Tried to move back new op lower bound from 7334629497152155648 to 7334629496961196032. Current Snapshot: MvccSnapshot[applied={T|T < 7334629497152155648}]
I20260929 11:23:49.606907 14814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:23:49.607193 14814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:23:49.610637 14814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Sending DeleteTablet for 3 replicas of tablet 9cf31c4c16b441f9a284d775eae70732
I20260929 11:23:49.611560 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf31c4c16b441f9a284d775eae70732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39496
I20260929 11:23:49.611584 15072 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf31c4c16b441f9a284d775eae70732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32768
I20260929 11:23:49.612120 15203 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf31c4c16b441f9a284d775eae70732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:23:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47736
I20260929 11:23:49.612211 15311 tablet_replica.cc:330] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: stopping tablet replica
I20260929 11:23:49.612466 15312 tablet_replica.cc:330] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: stopping tablet replica
I20260929 11:23:49.612490 15311 raft_consensus.cc:2229] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:23:49.612929 15312 raft_consensus.cc:2229] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:23:49.613168 15311 raft_consensus.cc:2258] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:49.613440 15312 raft_consensus.cc:2258] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:49.614197 15313 tablet_replica.cc:330] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: stopping tablet replica
I20260929 11:23:49.614581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14851
I20260929 11:23:49.614614 15311 ts_tablet_manager.cc:1905] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:49.614583 15313 raft_consensus.cc:2229] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:23:49.615144 15313 raft_consensus.cc:2258] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:23:49.617432 15312 ts_tablet_manager.cc:1905] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:49.617763 15313 ts_tablet_manager.cc:1905] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:23:49.621706 15311 ts_tablet_manager.cc:1918] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:49.621847 15311 log.cc:1196] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/9cf31c4c16b441f9a284d775eae70732
I20260929 11:23:49.622383 15311 ts_tablet_manager.cc:1939] T 9cf31c4c16b441f9a284d775eae70732 P 16564e3d08c94134af84fc187db0a520: Deleting consensus metadata
I20260929 11:23:49.623476 14798 catalog_manager.cc:4841] TS 16564e3d08c94134af84fc187db0a520 (127.0.97.1:42713): tablet 9cf31c4c16b441f9a284d775eae70732 (table test-table [id=fbc08931c5da4d6d9b1427af33ac4a43]) successfully deleted
I20260929 11:23:49.624207 15312 ts_tablet_manager.cc:1918] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:49.624366 15312 log.cc:1196] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/9cf31c4c16b441f9a284d775eae70732
I20260929 11:23:49.624953 15312 ts_tablet_manager.cc:1939] T 9cf31c4c16b441f9a284d775eae70732 P 2d50dc3297ef492e9ef7662bba3a63fc: Deleting consensus metadata
I20260929 11:23:49.626487 14801 catalog_manager.cc:4841] TS 2d50dc3297ef492e9ef7662bba3a63fc (127.0.97.2:33379): tablet 9cf31c4c16b441f9a284d775eae70732 (table test-table [id=fbc08931c5da4d6d9b1427af33ac4a43]) successfully deleted
I20260929 11:23:49.628952 15313 ts_tablet_manager.cc:1918] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:23:49.629102 15313 log.cc:1196] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/9cf31c4c16b441f9a284d775eae70732
I20260929 11:23:49.629562 15313 ts_tablet_manager.cc:1939] T 9cf31c4c16b441f9a284d775eae70732 P 836d7249ccc94daaa9432c1fc7b766bd: Deleting consensus metadata
I20260929 11:23:49.631283 14801 catalog_manager.cc:4841] TS 836d7249ccc94daaa9432c1fc7b766bd (127.0.97.3:35525): tablet 9cf31c4c16b441f9a284d775eae70732 (table test-table [id=fbc08931c5da4d6d9b1427af33ac4a43]) successfully deleted
I20260929 11:23:49.631994 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.615419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.0.97.1:42713)
0929 11:23:49.615601 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.615611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.616115 (+ 504us) server_negotiation.cc:407] Connection header received
0929 11:23:49.616173 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.616183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.616259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.616410 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:49.618164 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.619337 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.621256 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.621988 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.622143 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.625524 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:49.625535 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:49.625559 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:49.625594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:49.629079 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.629798 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.629805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.629807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.629946 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.630373 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.630376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.630378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.630707 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:49.631007 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.631471 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.631703 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:23:49.632709 14997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:49.731067 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14851
I20260929 11:23:49.750057 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15006
I20260929 11:23:49.763288 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.750925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.97.2:33379)
0929 11:23:49.751121 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.751127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.751153 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:23:49.751245 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.751253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.751340 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.751430 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:49.752843 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.753633 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.754834 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.755250 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.755347 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.758018 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:49.758030 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:49.758046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:49.758076 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:49.760775 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.761539 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.761547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.761549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.761674 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.761982 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.761985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.761987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.762323 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:49.762536 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.762741 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.762908 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:23:49.763926 15132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:49.853682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15006
I20260929 11:23:49.870718 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15140
I20260929 11:23:49.892799 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:49.875617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.0.97.3:35525)
0929 11:23:49.875816 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:23:49.875835 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:23:49.875931 (+ 96us) server_negotiation.cc:407] Connection header received
0929 11:23:49.876147 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:23:49.876153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:23:49.876225 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:23:49.876330 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:23:49.877833 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.878529 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.879865 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:23:49.880259 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:23:49.880393 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:23:49.887330 (+ 6937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:23:49.887343 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:23:49.887359 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:23:49.887396 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:23:49.890126 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.890807 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.890815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.890817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.890933 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:23:49.891321 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:23:49.891327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:23:49.891330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:23:49.891679 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:23:49.891976 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:23:49.892244 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 11:23:49.892475 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:23:49.893497 15263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:49.981305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15140
I20260929 11:23:49.998260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 14781
I20260929 11:23:49.999730 14844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:23:50.114537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 14781
I20260929 11:23:50.135701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38265
--webserver_interface=127.0.97.62
--webserver_port=36275
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:50.453116 15318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:50.453495 15318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:50.453665 15318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:50.460767 15318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:23:50.460916 15318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:50.460997 15318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:50.461076 15318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:23:50.461140 15318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:23:50.470758 15318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:50.472121 15318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:50.473481 15318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:50.485455 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:50.485534 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:50.485453 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:50.486353 15318 server_base.cc:1034] running on GCE node
I20260929 11:23:50.487190 15318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:50.488571 15318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:50.489833 15318 hybrid_clock.cc:648] HybridClock initialized: now 1790681030489772 us; error 52 us; skew 500 ppm
I20260929 11:23:50.490185 15318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:50 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:50.493243 15318 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:23:50.494626 15318 webserver.cc:460] Webserver started at http://127.0.97.62:36275/ using document root <none> and password file <none>
I20260929 11:23:50.495309 15318 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:50.495460 15318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:50.500792 15318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:23:50.503430 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:50.504848 15318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:23:50.505052 15318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "f040e582a6a5490f9e25fd428ba022e8"
format_stamp: "Formatted at 2026-09-29 11:23:44 on dist-test-slave-srst"
server_key: "904d6887dffeb784350f5c95c22ab60d"
server_key_iv: "f158138d1e16592449b2ecd1cfe1e44e"
server_key_version: "encryptionkey@0"
I20260929 11:23:50.506302 15318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:50.523146 15318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:50.527175 15318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:50.527475 15318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:50.545174 15318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38265
I20260929 11:23:50.545207 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38265 every 8 connection(s)
I20260929 11:23:50.546638 15318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:23:50.553390 15385 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:23:50.556372 15385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap starting.
I20260929 11:23:50.557027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15318
I20260929 11:23:50.557813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42713
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38265
--webserver_port=44365
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:50.576514 15385 log.cc:824] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:23:50.592069 15385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:23:50.592669 15385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8: Bootstrap complete.
I20260929 11:23:50.601621 15385 raft_consensus.cc:348] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:50.602460 15385 raft_consensus.cc:729] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f040e582a6a5490f9e25fd428ba022e8, State: Initialized, Role: FOLLOWER
I20260929 11:23:50.603066 15385 consensus_queue.cc:260] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:50.603292 15385 raft_consensus.cc:388] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:23:50.603435 15385 raft_consensus.cc:482] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:23:50.603645 15385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:23:50.607880 15385 raft_consensus.cc:504] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:50.608363 15385 leader_election.cc:302] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f040e582a6a5490f9e25fd428ba022e8; no voters:
I20260929 11:23:50.609067 15385 leader_election.cc:288] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 11:23:50.609195 15389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:23:50.609864 15389 raft_consensus.cc:686] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [term 3 LEADER]: Becoming Leader. State: Replica: f040e582a6a5490f9e25fd428ba022e8, State: Running, Role: LEADER
I20260929 11:23:50.610534 15389 consensus_queue.cc:237] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } }
I20260929 11:23:50.612351 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f040e582a6a5490f9e25fd428ba022e8. Latest consensus state: current_term: 3 leader_uuid: "f040e582a6a5490f9e25fd428ba022e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } } }
I20260929 11:23:50.612696 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:50.613238 15391 sys_catalog.cc:455] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f040e582a6a5490f9e25fd428ba022e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f040e582a6a5490f9e25fd428ba022e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38265 } } }
I20260929 11:23:50.613606 15391 sys_catalog.cc:458] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:23:50.617409 15385 sys_catalog.cc:564] T 00000000000000000000000000000000 P f040e582a6a5490f9e25fd428ba022e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:23:50.621731 15393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:23:50.629987 15393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:23:50.631832 15393 catalog_manager.cc:1259] Loaded cluster ID: 6336938a83794aec927b3d87caa0cda7
I20260929 11:23:50.631950 15393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 11:23:50.634471 15393 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb608f229ad google::LogMessage::Fail() at ??:0
@ 0x7fb608f2588a google::LogMessage::SendToLog() at ??:0
@ 0x7fb608f224de google::LogMessage::Flush() at ??:0
@ 0x7fb608f23a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb61e955512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb61e952976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb61e9da610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb61e9da439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb60bb6e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb60bb7191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb60bb71799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb60bb4ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb60ee266db start_thread at ??:0
@ 0x7fb607dbc61f clone at ??:0
W20260929 11:23:50.918455 15387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:50.918781 15387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:50.918936 15387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:50.925916 15387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:50.926064 15387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:50.926223 15387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:23:50.935607 15387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44365
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:50.936895 15387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:50.938235 15387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:50.950588 15426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:50.950629 15427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:50.950754 15429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:50.951218 15387 server_base.cc:1034] running on GCE node
I20260929 11:23:50.951808 15387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:50.952702 15387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:50.953893 15387 hybrid_clock.cc:648] HybridClock initialized: now 1790681030953846 us; error 39 us; skew 500 ppm
I20260929 11:23:50.954192 15387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:50 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:50.957170 15387 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:23:50.958619 15387 webserver.cc:460] Webserver started at http://127.0.97.1:44365/ using document root <none> and password file <none>
I20260929 11:23:50.959233 15387 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:50.959379 15387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:50.964491 15387 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:23:50.967028 15436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:50.968250 15387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 11:23:50.968413 15387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "16564e3d08c94134af84fc187db0a520"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "9e59de2633192efce0db75aae2b7934f"
server_key_iv: "6ba4277e72d7b670520592508d4b9dc3"
server_key_version: "encryptionkey@0"
I20260929 11:23:50.969447 15387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:50.989984 15387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:50.994180 15387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:50.994431 15387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:50.995352 15387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:50.996763 15387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:50.996881 15387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:50.997021 15387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:50.997102 15387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.030023 15387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42713
I20260929 11:23:51.030071 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42713 every 8 connection(s)
I20260929 11:23:51.031469 15387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:23:51.036103 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.034348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:46909)
0929 11:23:51.035047 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.035263 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":263,"threads_started":1}
W20260929 11:23:51.036772 15549 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38265 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38265: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
I20260929 11:23:51.037520 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.037202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:46423)
0929 11:23:51.037335 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.037414 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 11:23:51.038249 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.037983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:58815)
0929 11:23:51.038090 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.038154 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 11:23:51.038473 15549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:23:51.039810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15387
I20260929 11:23:51.040491 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33379
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38265
--webserver_port=38899
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:23:51.353425 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:51.353683 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:51.353840 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:51.360785 15552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:51.360934 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:51.361107 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:23:51.370463 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38899
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:51.371685 15552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:51.373072 15552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:51.385789 15557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:51.385740 15558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:51.386353 15560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:51.386389 15552 server_base.cc:1034] running on GCE node
I20260929 11:23:51.387010 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:51.387920 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:51.389135 15552 hybrid_clock.cc:648] HybridClock initialized: now 1790681031389090 us; error 54 us; skew 500 ppm
I20260929 11:23:51.389438 15552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:51 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:51.392462 15552 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:23:51.393934 15552 webserver.cc:460] Webserver started at http://127.0.97.2:38899/ using document root <none> and password file <none>
I20260929 11:23:51.394598 15552 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:51.394747 15552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:51.399909 15552 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:23:51.404359 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.405643 15552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:23:51.405825 15552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "2d50dc3297ef492e9ef7662bba3a63fc"
format_stamp: "Formatted at 2026-09-29 11:23:45 on dist-test-slave-srst"
server_key: "688bac4cd5633f0302e07bd4afa12e8a"
server_key_iv: "e34b60573c0c7bd53eec618c6730fd8a"
server_key_version: "encryptionkey@0"
I20260929 11:23:51.406838 15552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:51.418383 15552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:51.422457 15552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:51.422739 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:51.423573 15552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:51.424834 15552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:51.424949 15552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.425068 15552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:51.425150 15552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.454610 15552 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33379
I20260929 11:23:51.454695 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33379 every 8 connection(s)
I20260929 11:23:51.455997 15552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:23:51.460522 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.458835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:43457)
0929 11:23:51.459508 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.459691 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":256,"threads_started":1}
W20260929 11:23:51.461134 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38265 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38265: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
I20260929 11:23:51.462080 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.461796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:50863)
0929 11:23:51.461923 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.461983 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 11:23:51.462065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15552
I20260929 11:23:51.462837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35525
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38265
--webserver_port=33617
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:23:51.462873 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.462600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:37685)
0929 11:23:51.462708 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.462778 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 11:23:51.463099 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:23:51.770599 15683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:23:51.770901 15683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:23:51.771050 15683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:23:51.777882 15683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:23:51.778023 15683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:23:51.778185 15683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:23:51.787282 15683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33617
--tserver_master_addrs=127.0.97.62:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:23:51.788515 15683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:23:51.789745 15683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:23:51.802160 15688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:51.802186 15691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:23:51.802224 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:23:51.802963 15683 server_base.cc:1034] running on GCE node
I20260929 11:23:51.803478 15683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:23:51.804342 15683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:23:51.805565 15683 hybrid_clock.cc:648] HybridClock initialized: now 1790681031805517 us; error 53 us; skew 500 ppm
I20260929 11:23:51.805876 15683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:23:51 dist-test-slave-srst krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:23:51.808858 15683 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:23:51.810400 15683 webserver.cc:460] Webserver started at http://127.0.97.3:33617/ using document root <none> and password file <none>
I20260929 11:23:51.811091 15683 fs_manager.cc:359] Metadata directory not provided
I20260929 11:23:51.811221 15683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:23:51.816232 15683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:23:51.818819 15698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.820034 15683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:23:51.820158 15683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "836d7249ccc94daaa9432c1fc7b766bd"
format_stamp: "Formatted at 2026-09-29 11:23:46 on dist-test-slave-srst"
server_key: "6857097f065968a883e65011dcdd3ae5"
server_key_iv: "5497cc979807cc4bdca528a75a6d5546"
server_key_version: "encryptionkey@0"
I20260929 11:23:51.821122 15683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:23:51.835561 15683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:23:51.839315 15683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:23:51.839540 15683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:23:51.840397 15683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:23:51.841642 15683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:23:51.841723 15683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.841799 15683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:23:51.841871 15683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:23:51.874848 15683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35525
I20260929 11:23:51.874931 15810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35525 every 8 connection(s)
I20260929 11:23:51.876305 15683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:23:51.880920 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.879200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:59373)
0929 11:23:51.879874 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.880051 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":260,"threads_started":1}
W20260929 11:23:51.881495 15811 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38265 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38265: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
I20260929 11:23:51.882236 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.881946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:41413)
0929 11:23:51.882077 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.882141 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:23:51.883018 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:51.882688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:58801)
0929 11:23:51.882820 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:51.882895 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:23:51.883282 15811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:23:51.884096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15683
I20260929 11:23:52.039857 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:52.038995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:33795)
0929 11:23:52.039497 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:52.039605 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":233,"threads_started":1}
I20260929 11:23:52.464484 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:52.463751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:47391)
0929 11:23:52.464168 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:52.464269 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":200,"threads_started":1}
I20260929 11:23:52.884487 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:52.883797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:34543)
0929 11:23:52.884195 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:52.884277 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":197,"threads_started":1}
I20260929 11:23:53.041574 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:53.040859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:57739)
0929 11:23:53.041296 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:53.041395 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":202,"threads_started":1}
I20260929 11:23:53.466142 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:53.465341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:57121)
0929 11:23:53.465892 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:53.465972 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":280,"threads_started":1}
I20260929 11:23:53.885991 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:53.885315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:54439)
0929 11:23:53.885732 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:53.885817 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":208,"threads_started":1}
I20260929 11:23:54.043069 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:54.042380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:33821)
0929 11:23:54.042823 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:54.042898 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
I20260929 11:23:54.467772 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:54.467101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:43527)
0929 11:23:54.467520 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:54.467607 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260929 11:23:54.887547 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:54.886871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:42949)
0929 11:23:54.887310 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:54.887386 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260929 11:23:55.044858 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:55.043896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:52679)
0929 11:23:55.044424 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:55.044527 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":264,"threads_started":1}
I20260929 11:23:55.469504 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:55.468794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:45759)
0929 11:23:55.469231 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:55.469309 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260929 11:23:55.889269 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:55.888538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:47085)
0929 11:23:55.888998 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:55.889074 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260929 11:23:56.046667 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:56.045722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:58831)
0929 11:23:56.046306 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:56.046414 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":277,"threads_started":1}
I20260929 11:23:56.471103 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:56.470359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:50915)
0929 11:23:56.470860 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:56.470937 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260929 11:23:56.890688 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:56.890058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:34709)
0929 11:23:56.890472 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:56.890543 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260929 11:23:57.048460 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:57.047658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:41803)
0929 11:23:57.048154 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:57.048290 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260929 11:23:57.472741 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:57.471928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:36011)
0929 11:23:57.472407 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:57.472494 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":268,"threads_started":1}
I20260929 11:23:57.892253 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:57.891558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:60167)
0929 11:23:57.891982 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:57.892076 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260929 11:23:58.050441 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:58.049668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:52733)
0929 11:23:58.050142 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:58.050223 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":215,"threads_started":1}
I20260929 11:23:58.474481 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:58.473589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:46361)
0929 11:23:58.474192 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:58.474268 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":290,"threads_started":1}
I20260929 11:23:58.893884 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:58.893154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:59207)
0929 11:23:58.893603 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:58.893679 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":196,"threads_started":1}
I20260929 11:23:59.052037 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:59.051311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:51291)
0929 11:23:59.051809 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:59.051886 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":288,"threads_started":1}
I20260929 11:23:59.476028 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:59.475341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:60955)
0929 11:23:59.475800 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:59.475875 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260929 11:23:59.895617 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:23:59.894925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:34295)
0929 11:23:59.895359 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 11:23:59.895434 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260929 11:24:00.053563 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:00.052889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:43095)
0929 11:24:00.053318 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:00.053396 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":207,"threads_started":1}
I20260929 11:24:00.477597 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:00.476950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:55927)
0929 11:24:00.477366 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:00.477439 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":203,"threads_started":1}
I20260929 11:24:00.897337 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:00.896518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:39485)
0929 11:24:00.897081 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:00.897172 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":308,"threads_started":1}
I20260929 11:24:01.055367 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:01.054637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:49245)
0929 11:24:01.055106 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:01.055196 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
I20260929 11:24:01.479436 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:01.478673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:36731)
0929 11:24:01.479181 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:01.479271 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":275,"threads_started":1}
I20260929 11:24:01.899027 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:01.898268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:60961)
0929 11:24:01.898793 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:01.898869 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260929 11:24:02.057061 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:02.056315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:33731)
0929 11:24:02.056793 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:02.056887 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":209,"threads_started":1}
I20260929 11:24:02.481190 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:02.480431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:57885)
0929 11:24:02.480938 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:02.481016 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
I20260929 11:24:02.900722 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:02.899898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:49905)
0929 11:24:02.900396 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:02.900480 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
I20260929 11:24:03.058642 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:03.057877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:37733)
0929 11:24:03.058414 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:03.058494 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":343,"threads_started":1}
I20260929 11:24:03.482883 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:03.482188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:34539)
0929 11:24:03.482632 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:03.482710 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20260929 11:24:03.902236 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:03.901568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:53447)
0929 11:24:03.902002 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:03.902080 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260929 11:24:04.060251 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:04.059524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:51231)
0929 11:24:04.059982 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:04.060065 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
I20260929 11:24:04.484812 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:04.483948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:60155)
0929 11:24:04.484467 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:04.484555 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260929 11:24:04.903926 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:04.903176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:50797)
0929 11:24:04.903653 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:04.903742 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":239,"threads_started":1}
I20260929 11:24:05.061837 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:05.061122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:56913)
0929 11:24:05.061555 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:05.061626 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260929 11:24:05.486577 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:05.485813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:55037)
0929 11:24:05.486294 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:05.486378 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260929 11:24:05.905737 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:05.905001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:54397)
0929 11:24:05.905460 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:05.905546 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":230,"threads_started":1}
I20260929 11:24:06.063881 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:06.063033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.1:42941)
0929 11:24:06.063621 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:06.063693 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":374,"threads_started":1}
I20260929 11:24:06.488518 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:06.487818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.2:53177)
0929 11:24:06.488260 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:06.488345 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260929 11:24:06.894510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15387
I20260929 11:24:06.907624 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:06.906850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38265 (local address 127.0.97.3:53397)
0929 11:24:06.907356 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:06.907433 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
I20260929 11:24:06.912907 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:06.895682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.0.97.1:42713)
0929 11:24:06.896310 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:06.896328 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:06.896371 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:24:06.898612 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:06.898622 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:06.899053 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:06.899331 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:06.900703 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:06.901895 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:06.903085 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:06.903786 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:06.903916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:06.906501 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:06.906509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:06.906524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:06.906559 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:06.909590 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:06.910309 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:06.910319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:06.910332 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:06.910484 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:06.910852 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:06.910859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:06.910862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:06.911424 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:06.911715 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:06.912199 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 11:24:06.912650 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":303,"threads_started":1}
I20260929 11:24:06.914196 15544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:06.997007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15387
I20260929 11:24:07.014770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15552
I20260929 11:24:07.031561 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:07.015687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.97.2:33379)
0929 11:24:07.016182 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:07.016196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:07.016232 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:24:07.017810 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:07.017820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:07.018237 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:07.018521 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:07.019783 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:07.021088 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:07.022306 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:07.022957 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:07.023082 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:07.025765 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:07.025772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:07.025786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:07.025820 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:07.028470 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:07.029230 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:07.029236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:07.029242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:07.029397 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:07.029733 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:07.029736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:07.029738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:07.030239 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:07.030468 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:07.030871 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 11:24:07.031283 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260929 11:24:07.033013 15675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:07.116055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15552
I20260929 11:24:07.134279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15683
I20260929 11:24:07.150816 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:07.135342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.97.3:35525)
0929 11:24:07.135929 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:07.135943 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:07.135983 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:24:07.137608 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:07.137625 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:07.138044 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:07.138330 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:07.139505 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:07.140784 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:07.141960 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:07.142627 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:07.142757 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:07.145282 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:07.145289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:07.145305 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:07.145342 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:07.147874 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:07.148502 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:07.148508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:07.148514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:07.148735 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:07.149057 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:07.149061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:07.149063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:07.149547 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:07.149772 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:07.150104 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 11:24:07.150489 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":286,"threads_started":1}
I20260929 11:24:07.152213 15806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:07.235363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15683
2026-09-29T11:24:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24082 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:24:07 dist-test-slave-srst krb5kdc[15868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:24:07 dist-test-slave-srst krb5kdc[15868](info): set up 2 sockets
Sep 29 11:24:07 dist-test-slave-srst krb5kdc[15868](info): commencing operation
krb5kdc: starting...
W20260929 11:24:08.332860 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:24:08 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:24:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:24:08.554296 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46683
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:24:08.881800 15881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:08.882124 15881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:08.882300 15881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:08.889528 15881 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:24:08.889683 15881 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:24:08.889763 15881 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:24:08.889853 15881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:24:08.889943 15881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:08.890053 15881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:08.890154 15881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:24:08.890247 15881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:24:08.900164 15881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:08.901512 15881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:08.902941 15881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:08.915995 15886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:08.915886 15887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:08.916045 15881 server_base.cc:1034] running on GCE node
W20260929 11:24:08.915866 15889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:08.917182 15881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:08.918607 15881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:08.919850 15881 hybrid_clock.cc:648] HybridClock initialized: now 1790681048919788 us; error 39 us; skew 500 ppm
I20260929 11:24:08.920168 15881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:08 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:08.926188 15881 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:24:08.927562 15881 webserver.cc:460] Webserver started at http://127.0.97.62:34383/ using document root <none> and password file <none>
I20260929 11:24:08.928188 15881 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:08.928269 15881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:08.928529 15881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:24:08.931455 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "6b2427e3ba2a420ba3a205167b3c4921"
format_stamp: "Formatted at 2026-09-29 11:24:08 on dist-test-slave-srst"
server_key: "9b9643b59ba2f4c059bb85026cd34244"
server_key_iv: "39b90958b4f7e350c7c1b1a5b400c701"
server_key_version: "encryptionkey@0"
I20260929 11:24:08.932149 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b2427e3ba2a420ba3a205167b3c4921"
format_stamp: "Formatted at 2026-09-29 11:24:08 on dist-test-slave-srst"
server_key: "9b9643b59ba2f4c059bb85026cd34244"
server_key_iv: "39b90958b4f7e350c7c1b1a5b400c701"
server_key_version: "encryptionkey@0"
I20260929 11:24:08.937225 15881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:24:08.940431 15896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:08.941759 15881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:24:08.941937 15881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "6b2427e3ba2a420ba3a205167b3c4921"
format_stamp: "Formatted at 2026-09-29 11:24:08 on dist-test-slave-srst"
server_key: "9b9643b59ba2f4c059bb85026cd34244"
server_key_iv: "39b90958b4f7e350c7c1b1a5b400c701"
server_key_version: "encryptionkey@0"
I20260929 11:24:08.942132 15881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:08.959702 15881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:08.963438 15881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:08.963701 15881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:08.982168 15881 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46683
I20260929 11:24:08.982273 15947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46683 every 8 connection(s)
I20260929 11:24:08.983501 15881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:24:08.986248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15881
I20260929 11:24:08.986475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:24:08.987282 388 external_mini_cluster.cc:1428] Setting key b1bc699fb188deea7391af2846f9686e
I20260929 11:24:08.987771 15948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:09.000696 15948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap starting.
Sep 29 11:24:09 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:09.004498 15948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:09.005584 15948 log.cc:824] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:09.008222 15948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: No bootstrap required, opened a new log
I20260929 11:24:09.009867 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:08.989300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.97.62:46683)
0929 11:24:08.990032 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:08.990043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:08.990081 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:24:08.992233 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:08.992254 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:08.992693 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:08.993023 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:08.994768 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:08.996035 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:08.997436 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:08.998134 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:08.998385 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:09.002983 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:09.002990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:09.003007 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:09.003053 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:09.006100 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:09.006736 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:09.006742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:09.006748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:09.006935 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:09.007327 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:09.007331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:09.007333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:09.007797 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:09.008117 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:09.008493 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 11:24:09.008940 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
I20260929 11:24:09.016156 15948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:09.016520 15948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:09.016656 15948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2427e3ba2a420ba3a205167b3c4921, State: Initialized, Role: FOLLOWER
I20260929 11:24:09.017179 15948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:09.017405 15948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:24:09.017544 15948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:24:09.017701 15948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:09.019727 15948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:09.020187 15948 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2427e3ba2a420ba3a205167b3c4921; no voters:
I20260929 11:24:09.020787 15948 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:24:09.020913 15953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:24:09.021627 15953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2427e3ba2a420ba3a205167b3c4921, State: Running, Role: LEADER
I20260929 11:24:09.022250 15953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:09.023021 15948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:24:09.025000 15954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b2427e3ba2a420ba3a205167b3c4921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } } }
I20260929 11:24:09.024994 15955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2427e3ba2a420ba3a205167b3c4921. Latest consensus state: current_term: 1 leader_uuid: "6b2427e3ba2a420ba3a205167b3c4921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } } }
I20260929 11:24:09.025436 15954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: This master's current role is: LEADER
I20260929 11:24:09.025549 15955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: This master's current role is: LEADER
I20260929 11:24:09.030150 15958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:24:09.036551 15958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:24:09.047660 15958 catalog_manager.cc:1347] Generated new cluster ID: f9739324ad0646ef99071ae4aa46dcc1
I20260929 11:24:09.047797 15958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:24:09.066587 15958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:24:09.067831 15958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:24:09.073262 15958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Generated new TSK 0
I20260929 11:24:09.074061 15958 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:24:09.140733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46683
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:24:09.469722 15974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:09.470021 15974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:09.470185 15974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:09.477574 15974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:09.477718 15974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:09.477923 15974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:24:09.487696 15974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:09.489063 15974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:09.490427 15974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:09.503871 15980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:09.504362 15982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:09.504419 15979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:09.504575 15974 server_base.cc:1034] running on GCE node
I20260929 11:24:09.505322 15974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:09.506222 15974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:09.507436 15974 hybrid_clock.cc:648] HybridClock initialized: now 1790681049507377 us; error 35 us; skew 500 ppm
I20260929 11:24:09.507757 15974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:09 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:09.511124 15974 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:24:09.512627 15974 webserver.cc:460] Webserver started at http://127.0.97.1:35939/ using document root <none> and password file <none>
I20260929 11:24:09.513348 15974 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:09.513480 15974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:09.513823 15974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:24:09.516853 15974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "e49cba59f887407699d050cf8d8a86c6"
format_stamp: "Formatted at 2026-09-29 11:24:09 on dist-test-slave-srst"
server_key: "5807c96a5c37f83294e4c79c5c569643"
server_key_iv: "3afb1f0352042f7699ddf27d890dbdf3"
server_key_version: "encryptionkey@0"
I20260929 11:24:09.517632 15974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e49cba59f887407699d050cf8d8a86c6"
format_stamp: "Formatted at 2026-09-29 11:24:09 on dist-test-slave-srst"
server_key: "5807c96a5c37f83294e4c79c5c569643"
server_key_iv: "3afb1f0352042f7699ddf27d890dbdf3"
server_key_version: "encryptionkey@0"
I20260929 11:24:09.522761 15974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:24:09.526510 15989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:09.527849 15974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:24:09.528036 15974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "e49cba59f887407699d050cf8d8a86c6"
format_stamp: "Formatted at 2026-09-29 11:24:09 on dist-test-slave-srst"
server_key: "5807c96a5c37f83294e4c79c5c569643"
server_key_iv: "3afb1f0352042f7699ddf27d890dbdf3"
server_key_version: "encryptionkey@0"
I20260929 11:24:09.528229 15974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:09.548151 15974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:09.552773 15974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:09.553077 15974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:09.554131 15974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:09.555553 15974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:09.555661 15974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:09.555811 15974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:09.555884 15974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:09.588663 15974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36893
I20260929 11:24:09.588725 16101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36893 every 8 connection(s)
I20260929 11:24:09.589954 15974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:24:09.595522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 15974
I20260929 11:24:09.595752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:24:09.596407 388 external_mini_cluster.cc:1428] Setting key 722de340761dd218beceedb6767cbc69
Sep 29 11:24:09 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:09.610427 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:09.592412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44185 (local address 127.0.97.62:46683)
0929 11:24:09.592888 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:09.592892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:09.593862 (+ 970us) server_negotiation.cc:407] Connection header received
0929 11:24:09.595655 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:09.595661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:09.595751 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:09.595867 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:09.597561 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:09.598477 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:09.599988 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:09.600750 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:09.600894 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:09.604429 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:09.604438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:09.604446 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:09.604490 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:09.607232 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:09.608143 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:09.608147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:09.608149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:09.608257 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:09.608739 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:09.608742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:09.608744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:09.609011 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:09.609118 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:09.610019 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 11:24:09.610181 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260929 11:24:09.611939 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:09.593010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:44185)
0929 11:24:09.593707 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:09.593751 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:09.595396 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:09.596106 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:09.596114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:09.596718 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:09.597381 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:09.597401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:09.598647 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:09.598651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:09.599837 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:09.599848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:09.600943 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:09.600947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:09.601146 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:09.601922 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:09.601945 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:09.604220 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:09.607365 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:09.607376 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:09.607394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:09.608034 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:09.608357 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:09.608365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:09.608372 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:09.608630 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:09.609138 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:09.609149 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:09.609606 (+ 457us) client_negotiation.cc:769] Sending connection context
0929 11:24:09.609917 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 11:24:09.610491 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":275,"threads_started":1}
I20260929 11:24:09.613493 16102 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46683
I20260929 11:24:09.613814 16102 heartbeater.cc:464] Registering TS with master...
I20260929 11:24:09.614519 16102 heartbeater.cc:511] Master 127.0.97.62:46683 requested a full tablet report, sending...
I20260929 11:24:09.616546 15913 ts_manager.cc:194] Registered new tserver with Master: e49cba59f887407699d050cf8d8a86c6 (127.0.97.1:36893)
I20260929 11:24:09.618078 15913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44185
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:24:09.649792 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46683
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:24:09.967216 16108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:09.967470 16108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:09.967561 16108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:09.974534 16108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:09.974638 16108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:09.974749 16108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:24:09.986004 16108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:09.987430 16108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:09.989081 16108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:10.007596 16113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:10.007545 16114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:10.008216 16116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:10.009569 16108 server_base.cc:1034] running on GCE node
I20260929 11:24:10.010155 16108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:10.011082 16108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:10.012305 16108 hybrid_clock.cc:648] HybridClock initialized: now 1790681050012248 us; error 53 us; skew 500 ppm
I20260929 11:24:10.012661 16108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:10 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:10.015657 16108 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:24:10.017236 16108 webserver.cc:460] Webserver started at http://127.0.97.2:36377/ using document root <none> and password file <none>
I20260929 11:24:10.017952 16108 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:10.018121 16108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:10.018435 16108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:24:10.021379 16108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "a2771a73d7d24637929594cab380594e"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "8293c16653498d57d10e2e0bffb3b9e3"
server_key_iv: "3d9956c95ed8550f7d0565c2075c9847"
server_key_version: "encryptionkey@0"
I20260929 11:24:10.022117 16108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2771a73d7d24637929594cab380594e"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "8293c16653498d57d10e2e0bffb3b9e3"
server_key_iv: "3d9956c95ed8550f7d0565c2075c9847"
server_key_version: "encryptionkey@0"
I20260929 11:24:10.027347 16108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:24:10.030730 16123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:10.032203 16108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:24:10.032373 16108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "a2771a73d7d24637929594cab380594e"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "8293c16653498d57d10e2e0bffb3b9e3"
server_key_iv: "3d9956c95ed8550f7d0565c2075c9847"
server_key_version: "encryptionkey@0"
I20260929 11:24:10.032662 16108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:10.055557 16108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:10.059741 16108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:10.060070 16108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:10.061102 16108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:10.062412 16108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:10.062487 16108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:10.062652 16108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:10.062704 16108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:10.095728 16108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34017
I20260929 11:24:10.095821 16235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34017 every 8 connection(s)
I20260929 11:24:10.097543 16108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:24:10.104517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16108
I20260929 11:24:10.104770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:24:10.105257 388 external_mini_cluster.cc:1428] Setting key a8b9eb4c7963a77dfb240421d59993c9
Sep 29 11:24:10 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:10.118963 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.100713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34515 (local address 127.0.97.62:46683)
0929 11:24:10.100951 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.100958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.102198 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 11:24:10.104064 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.104069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.104163 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.104276 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:10.105933 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.106752 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.108411 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.109211 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.109354 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.112614 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:10.112623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:10.112631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:10.112674 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:10.115762 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.116461 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.116469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.116476 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.116665 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.117062 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.117070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.117076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.117439 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:10.117591 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.118243 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.118513 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:24:10.119693 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.101304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:34515)
0929 11:24:10.102024 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.102070 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.103761 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.104470 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.104478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.105117 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:10.105751 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.105776 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.106942 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.106947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.108265 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.108277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.109401 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.109406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.109604 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.110282 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:10.110304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:10.112371 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:10.115888 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:10.115894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:10.115907 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:10.116336 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:10.116780 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:10.116784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:10.116786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:10.116957 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:10.117581 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:10.117587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:10.117877 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.118106 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.118477 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260929 11:24:10.121268 16236 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46683
I20260929 11:24:10.121593 16236 heartbeater.cc:464] Registering TS with master...
I20260929 11:24:10.122289 16236 heartbeater.cc:511] Master 127.0.97.62:46683 requested a full tablet report, sending...
I20260929 11:24:10.123888 15913 ts_manager.cc:194] Registered new tserver with Master: a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
I20260929 11:24:10.125223 15913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34515
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:24:10.160352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46683
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:24:10.478030 16241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:10.478312 16241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:10.478518 16241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:10.485626 16241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:10.485734 16241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:10.485946 16241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:24:10.495441 16241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:10.496780 16241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:10.498150 16241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:10.511235 16246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:10.511266 16247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:10.512013 16249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:10.512419 16241 server_base.cc:1034] running on GCE node
I20260929 11:24:10.513007 16241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:10.513857 16241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:10.515062 16241 hybrid_clock.cc:648] HybridClock initialized: now 1790681050515017 us; error 40 us; skew 500 ppm
I20260929 11:24:10.515370 16241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:10 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:10.518487 16241 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:24:10.519778 16241 webserver.cc:460] Webserver started at http://127.0.97.3:43533/ using document root <none> and password file <none>
I20260929 11:24:10.520467 16241 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:10.520565 16241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:10.520874 16241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:24:10.523805 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "9253461dbdf54068ad7f95be33a5ef5a"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "5f0c422a62be30390a15433a886f9281"
server_key_iv: "b54da10fabc20242cf12588bcb198c49"
server_key_version: "encryptionkey@0"
I20260929 11:24:10.524672 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9253461dbdf54068ad7f95be33a5ef5a"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "5f0c422a62be30390a15433a886f9281"
server_key_iv: "b54da10fabc20242cf12588bcb198c49"
server_key_version: "encryptionkey@0"
I20260929 11:24:10.529454 16241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 11:24:10.533206 16256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:10.534549 16241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:24:10.534709 16241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "9253461dbdf54068ad7f95be33a5ef5a"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "5f0c422a62be30390a15433a886f9281"
server_key_iv: "b54da10fabc20242cf12588bcb198c49"
server_key_version: "encryptionkey@0"
I20260929 11:24:10.534914 16241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:10.557107 16241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:10.561568 16241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:10.561854 16241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:10.562835 16241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:10.564132 16241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:10.564235 16241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:10.564378 16241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:10.564461 16241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:10.597101 16241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41677
I20260929 11:24:10.597154 16368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41677 every 8 connection(s)
I20260929 11:24:10.598325 16241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:24:10.601514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16241
I20260929 11:24:10.601722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:24:10.602293 388 external_mini_cluster.cc:1428] Setting key 7526680048941a13203f6910a245b8ab
Sep 29 11:24:10 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:10.617774 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.600935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51867 (local address 127.0.97.62:46683)
0929 11:24:10.601114 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.601118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.602468 (+ 1350us) server_negotiation.cc:407] Connection header received
0929 11:24:10.604503 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.604508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.604606 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.604708 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:10.606134 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.606886 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.608316 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.608745 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.608834 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.612114 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:10.612119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:10.612122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:10.612151 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:10.614597 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.615238 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.615242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.615244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.615368 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.615866 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.615870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.615873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.616191 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:10.616327 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.617227 (+ 900us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.617437 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:24:10.618723 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.601443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:51867)
0929 11:24:10.602289 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.602337 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.604268 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.604868 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.604876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.605438 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:10.605987 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.606009 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.607020 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.607024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.608192 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.608203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.608911 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.608915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.609120 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.609769 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:10.609790 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:10.611910 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:10.614718 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:10.614724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:10.614735 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:10.615143 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:10.615480 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:10.615511 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:10.615517 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:10.615756 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:10.616319 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:10.616324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:10.616738 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.617086 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.617536 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":331,"threads_started":1}
I20260929 11:24:10.619890 16369 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46683
I20260929 11:24:10.620190 16369 heartbeater.cc:464] Registering TS with master...
I20260929 11:24:10.620885 16369 heartbeater.cc:511] Master 127.0.97.62:46683 requested a full tablet report, sending...
I20260929 11:24:10.621026 16102 heartbeater.cc:503] Master 127.0.97.62:46683 was elected leader, sending a full tablet report...
I20260929 11:24:10.622879 15913 ts_manager.cc:194] Registered new tserver with Master: 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677)
I20260929 11:24:10.623665 15913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51867
I20260929 11:24:10.629532 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:24:10.646984 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.634079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.97.62:46683)
0929 11:24:10.634274 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.634283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.634643 (+ 360us) server_negotiation.cc:407] Connection header received
0929 11:24:10.634839 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.634844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.634938 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.635040 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:10.636550 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.637307 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.638687 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.639088 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.639250 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.641641 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:10.641647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:10.641652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:10.641690 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:10.644257 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.645003 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.645010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.645014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.645168 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.645575 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.645581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.645584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.645874 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:10.646131 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.646464 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.646688 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:24:10.651324 15913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:24:10.658089 15913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:24:10.702790 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.670812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:33788)
0929 11:24:10.694010 (+ 23198us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.694032 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.694267 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.695046 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.695057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.695108 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:10.695606 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.695630 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.697935 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.697939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.699740 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.699756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.701678 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.701684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.701868 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.701919 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.702095 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.702276 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23051,"mutex_wait_us":83,"spinlock_wait_cycles":4352,"thread_start_us":21619,"threads_started":1}
I20260929 11:24:10.702912 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.669691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41677 (local address 127.0.0.1:44404)
0929 11:24:10.676433 (+ 6742us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.676486 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.677277 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.695428 (+ 18151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.695437 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.695473 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:10.696070 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.696082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.697745 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.697751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.700248 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.700260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.702090 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.702096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.702257 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.702285 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.702381 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.702511 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"mutex_wait_us":43,"thread_start_us":271,"threads_started":1}
I20260929 11:24:10.702925 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.669878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.0.97.3:41677)
0929 11:24:10.694886 (+ 25008us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.694894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.694926 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:24:10.695019 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.695026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.695239 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.695430 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:10.696269 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.697612 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.700434 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.701869 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.702004 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.702222 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.702434 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.702603 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24828,"thread_start_us":24696,"threads_started":1}
I20260929 11:24:10.703586 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.671492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.0.97.1:36893)
0929 11:24:10.691334 (+ 19842us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.691348 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.694158 (+ 2810us) server_negotiation.cc:407] Connection header received
0929 11:24:10.694531 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.694544 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.694807 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.695038 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:10.695807 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.697773 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.699937 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.701475 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.701636 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.701876 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.703252 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.703366 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19561,"thread_start_us":19869,"threads_started":1}
I20260929 11:24:10.704428 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.670369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34017 (local address 127.0.0.1:36764)
0929 11:24:10.692113 (+ 21744us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.692142 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.692323 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.695049 (+ 2726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.695057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.695137 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:10.695988 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.696006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.697868 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.697877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.700247 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.700260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.702395 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.702401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.702564 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.702599 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.702709 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.702838 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21559,"mutex_wait_us":408,"spinlock_wait_cycles":65920,"thread_start_us":23375,"threads_started":1}
I20260929 11:24:10.705217 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.670559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.97.2:34017)
0929 11:24:10.694325 (+ 23766us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.694333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.694361 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:24:10.694470 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.694481 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.694800 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.695432 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:10.696215 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.697749 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.700432 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.701869 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.702003 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.702191 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.704739 (+ 2548us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.704906 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23603,"thread_start_us":23517,"threads_started":1}
I20260929 11:24:10.707139 16037 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5bcbde204847b39db9df1ff1c7f3e0 (DEFAULT_TABLE table=test-table [id=be7233b395074797bfb8ae517586f39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:24:10.707605 16304 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5bcbde204847b39db9df1ff1c7f3e0 (DEFAULT_TABLE table=test-table [id=be7233b395074797bfb8ae517586f39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:24:10.708423 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5bcbde204847b39db9df1ff1c7f3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:10.708446 16171 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5bcbde204847b39db9df1ff1c7f3e0 (DEFAULT_TABLE table=test-table [id=be7233b395074797bfb8ae517586f39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:24:10.709017 16304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5bcbde204847b39db9df1ff1c7f3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:10.709748 16171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5bcbde204847b39db9df1ff1c7f3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:10.719520 16387 tablet_bootstrap.cc:492] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Bootstrap starting.
I20260929 11:24:10.722865 16387 tablet_bootstrap.cc:654] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:10.723960 16387 log.cc:824] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:10.726569 16387 tablet_bootstrap.cc:492] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: No bootstrap required, opened a new log
I20260929 11:24:10.726811 16387 ts_tablet_manager.cc:1397] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 11:24:10.729247 16389 tablet_bootstrap.cc:492] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Bootstrap starting.
I20260929 11:24:10.729499 16388 tablet_bootstrap.cc:492] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Bootstrap starting.
I20260929 11:24:10.732662 16388 tablet_bootstrap.cc:654] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:10.733362 16389 tablet_bootstrap.cc:654] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:10.733793 16388 log.cc:824] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:10.734520 16389 log.cc:824] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:10.737185 16388 tablet_bootstrap.cc:492] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: No bootstrap required, opened a new log
I20260929 11:24:10.737437 16388 ts_tablet_manager.cc:1397] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260929 11:24:10.737104 16387 raft_consensus.cc:348] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.737619 16387 raft_consensus.cc:374] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:10.737740 16387 raft_consensus.cc:729] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cba59f887407699d050cf8d8a86c6, State: Initialized, Role: FOLLOWER
I20260929 11:24:10.738507 16387 consensus_queue.cc:260] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.741999 16387 ts_tablet_manager.cc:1428] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260929 11:24:10.744689 16389 tablet_bootstrap.cc:492] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: No bootstrap required, opened a new log
I20260929 11:24:10.744958 16389 ts_tablet_manager.cc:1397] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260929 11:24:10.747108 16388 raft_consensus.cc:348] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.747447 16388 raft_consensus.cc:374] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:10.747548 16388 raft_consensus.cc:729] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9253461dbdf54068ad7f95be33a5ef5a, State: Initialized, Role: FOLLOWER
I20260929 11:24:10.748157 16388 consensus_queue.cc:260] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.749218 16369 heartbeater.cc:503] Master 127.0.97.62:46683 was elected leader, sending a full tablet report...
I20260929 11:24:10.749768 16388 ts_tablet_manager.cc:1428] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 11:24:10.752807 16389 raft_consensus.cc:348] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.753150 16389 raft_consensus.cc:374] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:10.753275 16389 raft_consensus.cc:729] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2771a73d7d24637929594cab380594e, State: Initialized, Role: FOLLOWER
I20260929 11:24:10.753796 16389 consensus_queue.cc:260] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.755134 16236 heartbeater.cc:503] Master 127.0.97.62:46683 was elected leader, sending a full tablet report...
I20260929 11:24:10.755594 16389 ts_tablet_manager.cc:1428] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 11:24:10.822803 16393 raft_consensus.cc:482] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:24:10.823145 16393 raft_consensus.cc:504] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.825176 16393 leader_election.cc:288] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
I20260929 11:24:10.831722 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.825764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50715 (local address 127.0.97.2:34017)
0929 11:24:10.825944 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.825949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.826569 (+ 620us) server_negotiation.cc:407] Connection header received
0929 11:24:10.826743 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.826748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.826825 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.826920 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:10.827544 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.828256 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.829978 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.830702 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.830859 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.830950 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.831322 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.831483 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:24:10.831734 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.825565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41055 (local address 127.0.97.3:41677)
0929 11:24:10.825758 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.825764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.826341 (+ 577us) server_negotiation.cc:407] Connection header received
0929 11:24:10.826539 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.826544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.826617 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.826705 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:10.827299 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.828098 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.829938 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.830669 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.830795 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.830917 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.831192 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.831338 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 11:24:10.832017 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.825462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34017 (local address 127.0.97.1:50715)
0929 11:24:10.826453 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.826472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.826596 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.826949 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.826951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.826970 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:10.827367 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.827380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.828393 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.828399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.829803 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.829814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.830955 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.830963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.831147 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.831179 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.831266 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.831378 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":24,"thread_start_us":642,"threads_started":1}
I20260929 11:24:10.832090 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.825376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41677 (local address 127.0.97.1:41055)
0929 11:24:10.826199 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:10.826220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:10.826386 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:10.826736 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:10.826741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:10.826772 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:10.827159 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.827169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.828227 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.828232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.829797 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:10.829808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.830890 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.830896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:10.831065 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.831090 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:24:10.831160 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 11:24:10.831263 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":42,"thread_start_us":428,"threads_started":1}
I20260929 11:24:10.833060 16324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bcbde204847b39db9df1ff1c7f3e0" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9253461dbdf54068ad7f95be33a5ef5a" is_pre_election: true
I20260929 11:24:10.833142 16191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bcbde204847b39db9df1ff1c7f3e0" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2771a73d7d24637929594cab380594e" is_pre_election: true
I20260929 11:24:10.833602 16191 raft_consensus.cc:2454] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 0.
I20260929 11:24:10.833541 16324 raft_consensus.cc:2454] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 0.
I20260929 11:24:10.834277 15991 leader_election.cc:302] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2771a73d7d24637929594cab380594e, e49cba59f887407699d050cf8d8a86c6; no voters:
I20260929 11:24:10.834653 16393 raft_consensus.cc:2781] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:24:10.834801 16393 raft_consensus.cc:482] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:24:10.834923 16393 raft_consensus.cc:3037] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:10.837399 16393 raft_consensus.cc:504] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
I20260929 11:24:10.838001 16393 leader_election.cc:288] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
I20260929 11:24:10.838426 16191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bcbde204847b39db9df1ff1c7f3e0" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2771a73d7d24637929594cab380594e"
I20260929 11:24:10.838445 16324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bcbde204847b39db9df1ff1c7f3e0" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9253461dbdf54068ad7f95be33a5ef5a"
I20260929 11:24:10.838634 16191 raft_consensus.cc:3037] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:10.838640 16324 raft_consensus.cc:3037] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:10.842026 16324 raft_consensus.cc:2454] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 1.
I20260929 11:24:10.842020 16191 raft_consensus.cc:2454] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 1.
W20260929 11:24:10.842427 16103 tablet.cc:2369] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:24:10.842468 15993 leader_election.cc:302] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9253461dbdf54068ad7f95be33a5ef5a, e49cba59f887407699d050cf8d8a86c6; no voters:
I20260929 11:24:10.842806 16393 raft_consensus.cc:2781] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:24:10.843366 16393 raft_consensus.cc:686] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 1 LEADER]: Becoming Leader. State: Replica: e49cba59f887407699d050cf8d8a86c6, State: Running, Role: LEADER
I20260929 11:24:10.844007 16393 consensus_queue.cc:237] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } }
W20260929 11:24:10.849808 16237 tablet.cc:2369] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:24:10.849279 15912 catalog_manager.cc:5495] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e49cba59f887407699d050cf8d8a86c6 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e49cba59f887407699d050cf8d8a86c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } health_report { overall_health: UNKNOWN } } }
W20260929 11:24:10.849908 16370 tablet.cc:2369] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:24:10.893956 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.888640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.0.97.1:36893)
0929 11:24:10.888856 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.888862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.888881 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:24:10.889030 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.889034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.889117 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.889278 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:24:10.889652 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.890356 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.891816 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.892219 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.892355 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.892875 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:24:10.893374 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:24:10.893544 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.893603 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.893695 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:24:10.902104 16324 raft_consensus.cc:1261] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Refusing update from remote peer e49cba59f887407699d050cf8d8a86c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:24:10.902096 16191 raft_consensus.cc:1261] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Refusing update from remote peer e49cba59f887407699d050cf8d8a86c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:24:10.903122 16398 consensus_queue.cc:1035] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:24:10.903429 16393 consensus_queue.cc:1035] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:24:10.916320 16403 mvcc.cc:205] Tried to move back new op lower bound from 7334629584484442112 to 7334629584261259264. Current Snapshot: MvccSnapshot[applied={T|T < 7334629584484442112}]
I20260929 11:24:10.928838 15913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:24:10.929147 15913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:24:10.932736 15913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Sending DeleteTablet for 3 replicas of tablet 1d5bcbde204847b39db9df1ff1c7f3e0
I20260929 11:24:10.933813 16304 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bcbde204847b39db9df1ff1c7f3e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:24:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44404
I20260929 11:24:10.934042 16037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bcbde204847b39db9df1ff1c7f3e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:24:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33788
I20260929 11:24:10.934412 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15974
I20260929 11:24:10.934525 16171 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bcbde204847b39db9df1ff1c7f3e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:24:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36764
I20260929 11:24:10.945508 16413 tablet_replica.cc:330] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: stopping tablet replica
I20260929 11:24:10.947403 16413 raft_consensus.cc:2229] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:24:10.947993 16413 raft_consensus.cc:2258] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:24:10.950675 16411 tablet_replica.cc:330] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: stopping tablet replica
I20260929 11:24:10.951187 16411 raft_consensus.cc:2229] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:24:10.951761 16412 tablet_replica.cc:330] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: stopping tablet replica
I20260929 11:24:10.951759 16411 raft_consensus.cc:2258] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:24:10.952145 16412 raft_consensus.cc:2229] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:24:10.953825 16411 ts_tablet_manager.cc:1905] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:24:10.954403 16413 ts_tablet_manager.cc:1905] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:24:10.952836 16412 raft_consensus.cc:2258] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:24:10.960928 16411 ts_tablet_manager.cc:1918] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:24:10.960928 16413 ts_tablet_manager.cc:1918] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:24:10.961077 16413 log.cc:1196] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/1d5bcbde204847b39db9df1ff1c7f3e0
I20260929 11:24:10.961583 16413 ts_tablet_manager.cc:1939] T 1d5bcbde204847b39db9df1ff1c7f3e0 P a2771a73d7d24637929594cab380594e: Deleting consensus metadata
I20260929 11:24:10.961607 16411 log.cc:1196] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/1d5bcbde204847b39db9df1ff1c7f3e0
I20260929 11:24:10.962180 16411 ts_tablet_manager.cc:1939] T 1d5bcbde204847b39db9df1ff1c7f3e0 P 9253461dbdf54068ad7f95be33a5ef5a: Deleting consensus metadata
I20260929 11:24:10.963042 16412 ts_tablet_manager.cc:1905] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:24:10.963804 15898 catalog_manager.cc:4841] TS a2771a73d7d24637929594cab380594e (127.0.97.2:34017): tablet 1d5bcbde204847b39db9df1ff1c7f3e0 (table test-table [id=be7233b395074797bfb8ae517586f39c]) successfully deleted
I20260929 11:24:10.964115 15900 catalog_manager.cc:4841] TS 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677): tablet 1d5bcbde204847b39db9df1ff1c7f3e0 (table test-table [id=be7233b395074797bfb8ae517586f39c]) successfully deleted
Sep 29 11:24:10 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:24:10.970283 16412 ts_tablet_manager.cc:1918] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:24:10.970454 16412 log.cc:1196] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/1d5bcbde204847b39db9df1ff1c7f3e0
I20260929 11:24:10.970999 16412 ts_tablet_manager.cc:1939] T 1d5bcbde204847b39db9df1ff1c7f3e0 P e49cba59f887407699d050cf8d8a86c6: Deleting consensus metadata
I20260929 11:24:10.972703 15900 catalog_manager.cc:4841] TS e49cba59f887407699d050cf8d8a86c6 (127.0.97.1:36893): tablet 1d5bcbde204847b39db9df1ff1c7f3e0 (table test-table [id=be7233b395074797bfb8ae517586f39c]) successfully deleted
I20260929 11:24:10.974673 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:10.935572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.97.1:36893)
0929 11:24:10.935799 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:10.935808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:10.948521 (+ 12713us) server_negotiation.cc:407] Connection header received
0929 11:24:10.948871 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:10.948879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:10.949009 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:10.949097 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:10.952957 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.957786 (+ 4829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.960165 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:10.960898 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:10.961004 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:10.968041 (+ 7037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:10.968056 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:10.968069 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:10.968118 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:10.971673 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.972381 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.972396 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.972399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.972616 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:10.973069 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:10.973074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:10.973078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:10.973421 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:10.973744 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:10.974086 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 11:24:10.974322 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:24:10.975428 16097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:11.074863 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15974
I20260929 11:24:11.094036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 16108
Sep 29 11:24:11 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:24:11.111825 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:11.094801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.0.97.2:34017)
0929 11:24:11.095036 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:11.095042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:11.095083 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:24:11.095157 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:11.095162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:11.095242 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:11.095366 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:11.096585 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:11.097739 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:11.099033 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:11.099718 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:11.099830 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:11.105312 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:11.105326 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:11.105344 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:11.105388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:11.109042 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:11.109831 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:11.109840 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:11.109844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:11.109974 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:11.110356 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:11.110361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:11.110365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:11.110809 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:11.111087 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:11.111223 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 11:24:11.111443 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:24:11.112541 16231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:11.200269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 16108
I20260929 11:24:11.214597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 16241
Sep 29 11:24:11 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:24:11.229202 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:11.215398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.0.97.3:41677)
0929 11:24:11.215589 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:11.215593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:11.215612 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:24:11.215784 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:11.215787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:11.215853 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:11.215938 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:11.217290 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:11.218021 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:11.219315 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:11.219698 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:11.219850 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:11.224169 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:11.224185 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:11.224201 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:11.224237 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:11.226920 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:11.227496 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:11.227503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:11.227505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:11.227654 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:11.227960 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:11.227963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:11.227964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:11.228289 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:11.228493 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:11.228798 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 11:24:11.228952 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:24:11.229822 16364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:11.316340 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 16241
I20260929 11:24:11.332429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 15881
I20260929 11:24:11.333686 15943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:11.441213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 15881
I20260929 11:24:31.462723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46683
--webserver_interface=127.0.97.62
--webserver_port=34383
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:24:31.778911 16419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:31.779175 16419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:31.779277 16419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:31.786209 16419 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:24:31.786293 16419 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:24:31.786330 16419 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:24:31.786373 16419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:24:31.786424 16419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:31.786489 16419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:31.786532 16419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:24:31.786579 16419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:24:31.795799 16419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:31.796929 16419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:31.798131 16419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:31.820499 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:31.820544 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:31.820565 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:31.822570 16419 server_base.cc:1034] running on GCE node
I20260929 11:24:31.823446 16419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:31.825055 16419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:31.826311 16419 hybrid_clock.cc:648] HybridClock initialized: now 1790681071826243 us; error 43 us; skew 500 ppm
I20260929 11:24:31.826715 16419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:31 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:31.835686 16419 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:24:31.837430 16419 webserver.cc:460] Webserver started at http://127.0.97.62:34383/ using document root <none> and password file <none>
I20260929 11:24:31.838181 16419 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:31.838284 16419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:31.844040 16419 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 11:24:31.846812 16435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:31.848114 16419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:24:31.848260 16419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "6b2427e3ba2a420ba3a205167b3c4921"
format_stamp: "Formatted at 2026-09-29 11:24:08 on dist-test-slave-srst"
server_key: "9b9643b59ba2f4c059bb85026cd34244"
server_key_iv: "39b90958b4f7e350c7c1b1a5b400c701"
server_key_version: "encryptionkey@0"
I20260929 11:24:31.849305 16419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:31.871392 16419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:31.875533 16419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:31.875783 16419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:31.894591 16419 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46683
I20260929 11:24:31.894681 16486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46683 every 8 connection(s)
I20260929 11:24:31.896107 16419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:24:31.903295 16487 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:24:31.904480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16419
I20260929 11:24:31.905259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36893
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46683
--webserver_port=35939
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:24:31.906977 16487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap starting.
I20260929 11:24:31.927104 16487 log.cc:824] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:31.939208 16487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:24:31.939755 16487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap complete.
I20260929 11:24:31.948794 16487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:31.949594 16487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2427e3ba2a420ba3a205167b3c4921, State: Initialized, Role: FOLLOWER
I20260929 11:24:31.950196 16487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:31.950438 16487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:24:31.950575 16487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:24:31.950776 16487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:24:31.954555 16487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:31.955039 16487 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2427e3ba2a420ba3a205167b3c4921; no voters:
I20260929 11:24:31.955699 16487 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:24:31.955823 16491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:24:31.958213 16487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:24:31.962960 16491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 LEADER]: Becoming Leader. State: Replica: 6b2427e3ba2a420ba3a205167b3c4921, State: Running, Role: LEADER
I20260929 11:24:31.963604 16491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:31.966245 16492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b2427e3ba2a420ba3a205167b3c4921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } } }
I20260929 11:24:31.966675 16492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: This master's current role is: LEADER
I20260929 11:24:31.967267 16493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2427e3ba2a420ba3a205167b3c4921. Latest consensus state: current_term: 2 leader_uuid: "6b2427e3ba2a420ba3a205167b3c4921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } } }
I20260929 11:24:31.967598 16493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: This master's current role is: LEADER
I20260929 11:24:31.977813 16502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:24:31.985790 16502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:24:31.987134 16502 catalog_manager.cc:1259] Loaded cluster ID: f9739324ad0646ef99071ae4aa46dcc1
I20260929 11:24:31.987236 16502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:24:31.991233 16502 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:24:31.992731 16502 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Loaded TSK: 0
I20260929 11:24:32.000974 16509 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:24:32.024338 16502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Generated new TSK 1
I20260929 11:24:32.024766 16502 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:24:32.260540 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:32.260951 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:32.261132 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:32.269644 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:32.269789 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:32.269968 16489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:24:32.279678 16489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35939
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:32.280975 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:32.282286 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:32.295702 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:32.295732 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:32.296911 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:32.296880 16489 server_base.cc:1034] running on GCE node
I20260929 11:24:32.297580 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:32.298575 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:32.299803 16489 hybrid_clock.cc:648] HybridClock initialized: now 1790681072299750 us; error 48 us; skew 500 ppm
I20260929 11:24:32.300115 16489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:32 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:32.303460 16489 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:24:32.305155 16489 webserver.cc:460] Webserver started at http://127.0.97.1:35939/ using document root <none> and password file <none>
I20260929 11:24:32.305851 16489 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:32.306010 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:32.311358 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 11:24:32.314158 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:32.315558 16489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:24:32.315728 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "e49cba59f887407699d050cf8d8a86c6"
format_stamp: "Formatted at 2026-09-29 11:24:09 on dist-test-slave-srst"
server_key: "5807c96a5c37f83294e4c79c5c569643"
server_key_iv: "3afb1f0352042f7699ddf27d890dbdf3"
server_key_version: "encryptionkey@0"
I20260929 11:24:32.316744 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:32.354812 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:32.359290 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:32.359565 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:32.360538 16489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:32.361896 16489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:32.362030 16489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:32.362159 16489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:32.362251 16489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:32.394493 16489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36893
I20260929 11:24:32.394578 16637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36893 every 8 connection(s)
I20260929 11:24:32.395884 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:24:32.398918 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16489
I20260929 11:24:32.399729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34017
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46683
--webserver_port=36377
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:24:32 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:32.421016 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:32.399037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:38921)
0929 11:24:32.399862 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:32.399929 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:32.402076 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:32.403510 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:32.403522 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:32.404122 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:32.404707 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:32.404726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.406321 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.406327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:32.407514 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:32.407526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.408533 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.408539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:32.408838 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:32.409502 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:32.409523 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:32.411803 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:32.416515 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:32.416522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:32.416535 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:32.417040 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:32.417567 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:32.417570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:32.417572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:32.417752 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:32.418753 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:32.418759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:32.419073 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 11:24:32.419304 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 11:24:32.419751 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":309,"threads_started":1}
I20260929 11:24:32.421442 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:32.400617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38921 (local address 127.0.97.62:46683)
0929 11:24:32.401207 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:32.401222 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:32.401276 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:24:32.402710 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:32.402723 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:32.403027 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:32.403380 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:32.404903 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.406165 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.407672 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.408309 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.408504 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:32.412077 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:32.412089 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:32.412110 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:32.412167 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:32.416353 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:32.417182 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:32.417195 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:32.417206 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:32.417440 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:32.417881 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:32.417889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:32.417895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:32.418607 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:32.418819 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:32.419606 (+ 787us) server_negotiation.cc:299] Negotiation successful
0929 11:24:32.420078 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260929 11:24:32.424847 16638 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46683
I20260929 11:24:32.425181 16638 heartbeater.cc:464] Registering TS with master...
I20260929 11:24:32.425930 16638 heartbeater.cc:511] Master 127.0.97.62:46683 requested a full tablet report, sending...
I20260929 11:24:32.427985 16452 ts_manager.cc:194] Registered new tserver with Master: e49cba59f887407699d050cf8d8a86c6 (127.0.97.1:36893)
I20260929 11:24:32.429682 16452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38921
W20260929 11:24:32.720211 16642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:32.720518 16642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:32.720726 16642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:32.727613 16642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:32.727751 16642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:32.727937 16642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:24:32.737308 16642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36377
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:32.738545 16642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:32.739789 16642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:32.752905 16647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:32.752857 16650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:32.752940 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:32.753813 16642 server_base.cc:1034] running on GCE node
I20260929 11:24:32.754396 16642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:32.755183 16642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:32.756386 16642 hybrid_clock.cc:648] HybridClock initialized: now 1790681072756344 us; error 30 us; skew 500 ppm
I20260929 11:24:32.756743 16642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:32 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:32.759933 16642 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:24:32.761430 16642 webserver.cc:460] Webserver started at http://127.0.97.2:36377/ using document root <none> and password file <none>
I20260929 11:24:32.762086 16642 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:32.762253 16642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:32.767740 16642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 11:24:32.770457 16657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:32.771838 16642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:24:32.772022 16642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "a2771a73d7d24637929594cab380594e"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "8293c16653498d57d10e2e0bffb3b9e3"
server_key_iv: "3d9956c95ed8550f7d0565c2075c9847"
server_key_version: "encryptionkey@0"
I20260929 11:24:32.773128 16642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:32.795392 16642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:32.799607 16642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:32.799901 16642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:32.800930 16642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:32.802228 16642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:32.802364 16642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:32.802508 16642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:32.802601 16642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:32.835602 16642 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34017
I20260929 11:24:32.835691 16769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34017 every 8 connection(s)
I20260929 11:24:32.837117 16642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:24:32.844205 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16642
I20260929 11:24:32.844892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41677
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46683
--webserver_port=43533
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:24:32 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:32.856499 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:32.839603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56035 (local address 127.0.97.62:46683)
0929 11:24:32.839777 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:32.839781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:32.840913 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 11:24:32.842761 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:32.842767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:32.842863 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:32.842973 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:32.844531 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.845410 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.846917 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.847336 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.847469 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:32.850867 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:32.850872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:32.850875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:32.850908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:32.853609 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:32.854308 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:32.854312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:32.854315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:32.854432 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:32.854822 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:32.854828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:32.854831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:32.855107 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:32.855259 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:32.855923 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 11:24:32.856143 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:24:32.857471 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:32.840026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:56035)
0929 11:24:32.840758 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:32.840794 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:32.842477 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:32.843231 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:32.843239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:32.843760 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:32.844320 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:32.844374 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.845566 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.845579 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:32.846766 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:32.846777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:32.847563 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:32.847570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:32.847806 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:32.848474 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:32.848500 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:32.850633 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:32.853742 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:32.853750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:32.853766 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:32.854186 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:32.854536 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:32.854540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:32.854544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:32.854715 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:32.855232 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:32.855240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:32.855564 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 11:24:32.855837 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 11:24:32.856356 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":275,"threads_started":1}
I20260929 11:24:32.858930 16770 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46683
I20260929 11:24:32.859272 16770 heartbeater.cc:464] Registering TS with master...
I20260929 11:24:32.860143 16770 heartbeater.cc:511] Master 127.0.97.62:46683 requested a full tablet report, sending...
I20260929 11:24:32.861840 16452 ts_manager.cc:194] Registered new tserver with Master: a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
I20260929 11:24:32.863195 16452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56035
W20260929 11:24:33.169309 16773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:33.169566 16773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:33.169659 16773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:33.176751 16773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:33.176884 16773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:33.177031 16773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:24:33.187551 16773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43533
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:33.188769 16773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:33.190025 16773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:33.202517 16778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:33.202435 16779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:33.202546 16781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:33.203555 16773 server_base.cc:1034] running on GCE node
I20260929 11:24:33.204123 16773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:33.205071 16773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:33.206287 16773 hybrid_clock.cc:648] HybridClock initialized: now 1790681073206242 us; error 62 us; skew 500 ppm
I20260929 11:24:33.206583 16773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:33 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:33.209545 16773 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:24:33.210846 16773 webserver.cc:460] Webserver started at http://127.0.97.3:43533/ using document root <none> and password file <none>
I20260929 11:24:33.211452 16773 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:33.211557 16773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:33.216727 16773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:24:33.219183 16788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:33.220525 16773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:24:33.220750 16773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "9253461dbdf54068ad7f95be33a5ef5a"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "5f0c422a62be30390a15433a886f9281"
server_key_iv: "b54da10fabc20242cf12588bcb198c49"
server_key_version: "encryptionkey@0"
I20260929 11:24:33.221709 16773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:33.236541 16773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:33.240221 16773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:33.240509 16773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:33.241431 16773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:33.242642 16773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:33.242725 16773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:33.242871 16773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:33.242981 16773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:33.276064 16773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41677
I20260929 11:24:33.276146 16900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41677 every 8 connection(s)
I20260929 11:24:33.277623 16773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:24:33.278811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16773
Sep 29 11:24:33 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:24:33.295697 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.279841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.97.62:46683)
0929 11:24:33.280052 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.280060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.280551 (+ 491us) server_negotiation.cc:407] Connection header received
0929 11:24:33.280808 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.280812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.280872 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.280980 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:33.282990 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.283716 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.285678 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.286145 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.286288 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.289530 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:33.289536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:33.289541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:33.289576 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:33.292735 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.293502 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.293505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.293508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.293636 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.294054 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.294060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.294063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.294431 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:33.294745 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.294969 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.295257 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:24:33.300307 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.280385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44833 (local address 127.0.97.62:46683)
0929 11:24:33.280858 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.280863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.282063 (+ 1200us) server_negotiation.cc:407] Connection header received
0929 11:24:33.284242 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.284246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.284296 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.284365 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:33.286161 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.286987 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.289414 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.290124 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.290268 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.294121 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:33.294128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:33.294136 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:33.294169 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:33.297389 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.298182 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.298194 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.298196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.298295 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.298691 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.298696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.298699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.298933 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:33.299085 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.299709 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.299930 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":278,"threads_started":1}
I20260929 11:24:33.301362 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.281069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:44833)
0929 11:24:33.281894 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.281950 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:33.283978 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:33.284665 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:33.284677 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:33.285343 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:33.286014 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.286030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.287138 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.287145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.289262 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.289288 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.290357 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.290365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.290666 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.291259 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:33.291286 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:33.293895 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:33.297518 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:33.297526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:33.297540 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:33.298065 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:33.298397 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:33.298402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:33.298406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:33.298578 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:33.299053 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:33.299060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:33.299364 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 11:24:33.299612 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 11:24:33.300095 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":298,"threads_started":1}
I20260929 11:24:33.302551 16901 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46683
I20260929 11:24:33.302810 16901 heartbeater.cc:464] Registering TS with master...
I20260929 11:24:33.303462 16901 heartbeater.cc:511] Master 127.0.97.62:46683 requested a full tablet report, sending...
I20260929 11:24:33.304797 16452 ts_manager.cc:194] Registered new tserver with Master: 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677)
I20260929 11:24:33.305518 16452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44833
I20260929 11:24:33.310317 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:24:33.316324 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.311184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.0.97.62:46683)
0929 11:24:33.311333 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.311337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.311833 (+ 496us) server_negotiation.cc:407] Connection header received
0929 11:24:33.311999 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.312002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.312125 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.312240 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:24:33.312649 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.313359 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.314769 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.315158 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.315247 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.315667 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:24:33.315886 (+ 219us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 11:24:33.316129 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52312: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 11:24:33.316463 16904 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52312: token expired
W20260929 11:24:33.316969 16373 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46683)
I20260929 11:24:33.329506 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.317385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.0.97.62:46683)
0929 11:24:33.317541 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.317547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.317593 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:24:33.317761 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.317768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.317828 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.317924 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:33.319188 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.319836 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.321528 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.321927 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.322058 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.324474 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:33.324479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:33.324482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:33.324509 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:33.327012 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.327622 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.327628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.327631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.327784 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.328146 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.328151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.328154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.328426 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:33.328708 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.328974 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.329173 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:24:33.332916 16452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:24:33.340480 16452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:24:33.366506 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.351656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41677 (local address 127.0.0.1:46120)
0929 11:24:33.352614 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.352655 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:33.352867 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:33.354770 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:33.354782 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:33.354828 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:33.355464 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.355492 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.358992 (+ 3500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.359002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.362497 (+ 3495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.362520 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.365402 (+ 2882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.365412 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.365615 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.365679 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 11:24:33.365804 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 11:24:33.366030 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":429,"threads_started":1}
I20260929 11:24:33.367290 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.351966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.0.97.3:41677)
0929 11:24:33.353931 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.353945 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.353991 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:24:33.354119 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.354132 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.354444 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.356102 (+ 1658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:33.356507 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.358781 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.362915 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.365129 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.366482 (+ 1353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.366726 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.366796 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.366978 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":1476,"threads_started":1}
I20260929 11:24:33.371268 16836 tablet_service.cc:1467] Processing CreateTablet for tablet 060b26f1de8249c69c88fe1818a8d1d2 (DEFAULT_TABLE table=test-table [id=d44df1fb49344aeab283721fedb81667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:24:33.372920 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060b26f1de8249c69c88fe1818a8d1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:33.372917 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.351839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34017 (local address 127.0.0.1:46142)
0929 11:24:33.363831 (+ 11992us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.363853 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:33.363987 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:33.366015 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:33.366020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:33.366047 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:33.366630 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.366642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.368709 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.368714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.370460 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.370471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.371897 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.371955 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.372138 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.372178 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:24:33.372302 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 11:24:33.372464 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11841,"mutex_wait_us":148,"thread_start_us":12233,"threads_started":1}
I20260929 11:24:33.373014 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.352556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.97.2:34017)
0929 11:24:33.365382 (+ 12826us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.365392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.365425 (+ 33us) server_negotiation.cc:407] Connection header received
0929 11:24:33.365525 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.365535 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.365789 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.366345 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:33.366836 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.368297 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.370604 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.371667 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.371800 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.371975 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.372352 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.372552 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12623,"thread_start_us":13221,"threads_started":1}
I20260929 11:24:33.374424 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.353485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50456)
0929 11:24:33.365148 (+ 11663us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.365164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:33.365275 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:33.366221 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:33.366225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:33.366265 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:33.367224 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.367234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.368981 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.368986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.371138 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.371153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.373521 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.373528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.373732 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.373769 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:24:33.373870 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 11:24:33.373992 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11540,"mutex_wait_us":586,"spinlock_wait_cycles":6528,"thread_start_us":11392,"threads_started":1}
I20260929 11:24:33.374503 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.354050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.0.97.1:36893)
0929 11:24:33.364683 (+ 10633us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.364695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.365387 (+ 692us) server_negotiation.cc:407] Connection header received
0929 11:24:33.365594 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.365602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.365842 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.366042 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:33.367487 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.368796 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.371315 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.372888 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.373023 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.373215 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.374011 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.374173 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10451,"thread_start_us":10348,"threads_started":1}
I20260929 11:24:33.376513 16705 tablet_service.cc:1467] Processing CreateTablet for tablet 060b26f1de8249c69c88fe1818a8d1d2 (DEFAULT_TABLE table=test-table [id=d44df1fb49344aeab283721fedb81667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:24:33.377840 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060b26f1de8249c69c88fe1818a8d1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:33.379191 16573 tablet_service.cc:1467] Processing CreateTablet for tablet 060b26f1de8249c69c88fe1818a8d1d2 (DEFAULT_TABLE table=test-table [id=d44df1fb49344aeab283721fedb81667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:24:33.380424 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060b26f1de8249c69c88fe1818a8d1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:24:33.387290 16917 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Bootstrap starting.
I20260929 11:24:33.390599 16917 tablet_bootstrap.cc:654] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:33.391829 16917 log.cc:824] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:33.394919 16917 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: No bootstrap required, opened a new log
I20260929 11:24:33.395264 16917 ts_tablet_manager.cc:1397] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260929 11:24:33.401665 16919 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Bootstrap starting.
I20260929 11:24:33.401970 16920 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Bootstrap starting.
I20260929 11:24:33.405200 16920 tablet_bootstrap.cc:654] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:33.405197 16919 tablet_bootstrap.cc:654] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Neither blocks nor log segments found. Creating new log.
I20260929 11:24:33.406423 16920 log.cc:824] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:33.406440 16919 log.cc:824] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:33.406630 16917 raft_consensus.cc:348] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.407133 16917 raft_consensus.cc:374] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:33.407291 16917 raft_consensus.cc:729] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9253461dbdf54068ad7f95be33a5ef5a, State: Initialized, Role: FOLLOWER
I20260929 11:24:33.408088 16917 consensus_queue.cc:260] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.409309 16901 heartbeater.cc:503] Master 127.0.97.62:46683 was elected leader, sending a full tablet report...
I20260929 11:24:33.410020 16917 ts_tablet_manager.cc:1428] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260929 11:24:33.410146 16919 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: No bootstrap required, opened a new log
I20260929 11:24:33.410604 16919 ts_tablet_manager.cc:1397] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:24:33.410888 16920 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: No bootstrap required, opened a new log
I20260929 11:24:33.411172 16920 ts_tablet_manager.cc:1397] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260929 11:24:33.418619 16920 raft_consensus.cc:348] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.419001 16920 raft_consensus.cc:374] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:33.418838 16919 raft_consensus.cc:348] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.419106 16920 raft_consensus.cc:729] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cba59f887407699d050cf8d8a86c6, State: Initialized, Role: FOLLOWER
I20260929 11:24:33.419247 16919 raft_consensus.cc:374] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:24:33.419341 16919 raft_consensus.cc:729] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2771a73d7d24637929594cab380594e, State: Initialized, Role: FOLLOWER
I20260929 11:24:33.419740 16920 consensus_queue.cc:260] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.419811 16919 consensus_queue.cc:260] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.420933 16638 heartbeater.cc:503] Master 127.0.97.62:46683 was elected leader, sending a full tablet report...
I20260929 11:24:33.421470 16920 ts_tablet_manager.cc:1428] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 11:24:33.421552 16770 heartbeater.cc:503] Master 127.0.97.62:46683 was elected leader, sending a full tablet report...
I20260929 11:24:33.422102 16919 ts_tablet_manager.cc:1428] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20260929 11:24:33.529512 16902 tablet.cc:2369] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:24:33.589236 16771 tablet.cc:2369] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:24:33.609531 16924 raft_consensus.cc:482] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:24:33.609848 16924 raft_consensus.cc:504] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.611518 16924 leader_election.cc:288] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
I20260929 11:24:33.618113 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.612026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55709 (local address 127.0.97.2:34017)
0929 11:24:33.612267 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.612276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.612904 (+ 628us) server_negotiation.cc:407] Connection header received
0929 11:24:33.613136 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.613141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.613201 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.613299 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:33.613836 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.614539 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.616290 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.617138 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.617282 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.617373 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.617731 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.617857 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:24:33.618338 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.611933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54681 (local address 127.0.97.3:41677)
0929 11:24:33.612202 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.612211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.612942 (+ 731us) server_negotiation.cc:407] Connection header received
0929 11:24:33.613177 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.613181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.613237 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.613326 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:24:33.613955 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.614654 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.616352 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.617181 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.617322 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.617427 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.617962 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.618087 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:24:33.618363 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.611770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34017 (local address 127.0.97.1:55709)
0929 11:24:33.612790 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.612810 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:33.612980 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:33.613312 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:33.613317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:33.613339 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:33.613698 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.613707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.614669 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.614675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.616143 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.616154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.617358 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.617364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.617541 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.617603 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 11:24:33.617678 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 11:24:33.617764 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":22,"thread_start_us":665,"threads_started":1}
I20260929 11:24:33.618475 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.611878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41677 (local address 127.0.97.1:54681)
0929 11:24:33.612827 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.612841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:33.613027 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:33.613351 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:33.613356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:33.613379 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:24:33.613791 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.613799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.614778 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.614785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.616199 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:33.616211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.617548 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.617552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:33.617788 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.617821 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:24:33.617915 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 11:24:33.618038 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":41,"spinlock_wait_cycles":29440,"thread_start_us":615,"threads_started":1}
I20260929 11:24:33.619421 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060b26f1de8249c69c88fe1818a8d1d2" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2771a73d7d24637929594cab380594e" is_pre_election: true
I20260929 11:24:33.619421 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060b26f1de8249c69c88fe1818a8d1d2" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9253461dbdf54068ad7f95be33a5ef5a" is_pre_election: true
I20260929 11:24:33.619962 16725 raft_consensus.cc:2454] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 0.
I20260929 11:24:33.619968 16856 raft_consensus.cc:2454] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 0.
I20260929 11:24:33.620699 16529 leader_election.cc:302] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9253461dbdf54068ad7f95be33a5ef5a, a2771a73d7d24637929594cab380594e, e49cba59f887407699d050cf8d8a86c6; no voters:
I20260929 11:24:33.621093 16924 raft_consensus.cc:2781] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:24:33.621241 16924 raft_consensus.cc:482] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:24:33.621344 16924 raft_consensus.cc:3037] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:33.623458 16924 raft_consensus.cc:504] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.624048 16924 leader_election.cc:288] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
I20260929 11:24:33.624511 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060b26f1de8249c69c88fe1818a8d1d2" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2771a73d7d24637929594cab380594e"
I20260929 11:24:33.624554 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060b26f1de8249c69c88fe1818a8d1d2" candidate_uuid: "e49cba59f887407699d050cf8d8a86c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9253461dbdf54068ad7f95be33a5ef5a"
I20260929 11:24:33.624749 16725 raft_consensus.cc:3037] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:33.624811 16856 raft_consensus.cc:3037] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:24:33.626905 16725 raft_consensus.cc:2454] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 1.
I20260929 11:24:33.627252 16856 raft_consensus.cc:2454] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cba59f887407699d050cf8d8a86c6 in term 1.
I20260929 11:24:33.627270 16527 leader_election.cc:302] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2771a73d7d24637929594cab380594e, e49cba59f887407699d050cf8d8a86c6; no voters:
I20260929 11:24:33.627568 16924 raft_consensus.cc:2781] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:24:33.628031 16924 raft_consensus.cc:686] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 LEADER]: Becoming Leader. State: Replica: e49cba59f887407699d050cf8d8a86c6, State: Running, Role: LEADER
I20260929 11:24:33.628489 16924 consensus_queue.cc:237] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:33.633147 16452 catalog_manager.cc:5495] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e49cba59f887407699d050cf8d8a86c6 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e49cba59f887407699d050cf8d8a86c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } health_report { overall_health: HEALTHY } } }
W20260929 11:24:33.648185 16639 tablet.cc:2369] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:24:33.666848 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.661618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.0.97.1:36893)
0929 11:24:33.661823 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.661829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.661847 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:24:33.661962 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.661965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.662035 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.662196 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:24:33.662555 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.663244 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.664728 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.665132 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.665264 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.665790 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:24:33.666261 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:24:33.666419 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.666500 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.666576 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:24:33.674542 16856 raft_consensus.cc:1261] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Refusing update from remote peer e49cba59f887407699d050cf8d8a86c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:24:33.674540 16725 raft_consensus.cc:1261] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Refusing update from remote peer e49cba59f887407699d050cf8d8a86c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:24:33.675514 16928 consensus_queue.cc:1035] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:24:33.675835 16924 consensus_queue.cc:1035] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:24:33.686748 16930 mvcc.cc:205] Tried to move back new op lower bound from 7334629677761130496 to 7334629677585735680. Current Snapshot: MvccSnapshot[applied={T|T < 7334629677761130496}]
I20260929 11:24:33.700647 16451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:24:33.700956 16451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:24:33.704460 16451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Sending DeleteTablet for 3 replicas of tablet 060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:24:33.705615 16705 tablet_service.cc:1514] Processing DeleteTablet for tablet 060b26f1de8249c69c88fe1818a8d1d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:24:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260929 11:24:33.705729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 16489
I20260929 11:24:33.705695 16836 tablet_service.cc:1514] Processing DeleteTablet for tablet 060b26f1de8249c69c88fe1818a8d1d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:24:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46120
I20260929 11:24:33.706563 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet 060b26f1de8249c69c88fe1818a8d1d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:24:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50456
I20260929 11:24:33.729321 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.707269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.0.97.1:36893)
0929 11:24:33.707463 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.707471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.707491 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:24:33.707550 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.707557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.707653 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.707755 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:33.710025 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.711481 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.713896 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.714804 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.714944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.719393 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:33.719409 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:33.719431 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:33.719497 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:33.724223 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.725194 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.725205 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.725282 (+ 77us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.725486 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.726025 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.726033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.726038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.726732 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:33.727175 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.728483 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.728776 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:24:33.730016 16633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:33.731475 16941 tablet_replica.cc:330] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: stopping tablet replica
I20260929 11:24:33.732023 16941 raft_consensus.cc:2229] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:24:33.732735 16941 raft_consensus.cc:2258] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:24:33.733207 16940 tablet_replica.cc:330] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: stopping tablet replica
I20260929 11:24:33.733635 16940 raft_consensus.cc:2229] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:24:33.734161 16940 raft_consensus.cc:2258] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:24:33.735525 16941 ts_tablet_manager.cc:1905] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:24:33.736075 16940 ts_tablet_manager.cc:1905] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:24:33.742321 16941 ts_tablet_manager.cc:1918] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:24:33.742506 16941 log.cc:1196] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:24:33.743040 16941 ts_tablet_manager.cc:1939] T 060b26f1de8249c69c88fe1818a8d1d2 P 9253461dbdf54068ad7f95be33a5ef5a: Deleting consensus metadata
I20260929 11:24:33.743069 16940 ts_tablet_manager.cc:1918] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:24:33.743216 16940 log.cc:1196] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:24:33.743712 16940 ts_tablet_manager.cc:1939] T 060b26f1de8249c69c88fe1818a8d1d2 P a2771a73d7d24637929594cab380594e: Deleting consensus metadata
I20260929 11:24:33.744910 16439 catalog_manager.cc:4841] TS 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677): tablet 060b26f1de8249c69c88fe1818a8d1d2 (table test-table [id=d44df1fb49344aeab283721fedb81667]) successfully deleted
I20260929 11:24:33.745326 16437 catalog_manager.cc:4841] TS a2771a73d7d24637929594cab380594e (127.0.97.2:34017): tablet 060b26f1de8249c69c88fe1818a8d1d2 (table test-table [id=d44df1fb49344aeab283721fedb81667]) successfully deleted
I20260929 11:24:33.829998 16942 tablet_replica.cc:330] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: stopping tablet replica
I20260929 11:24:33.830217 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 16489
I20260929 11:24:33.853190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 16642
W20260929 11:24:33.853319 16439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36893 (error 108)
I20260929 11:24:33.855885 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.855470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50482)
0929 11:24:33.855646 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.855746 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 11:24:33.856199 16439 catalog_manager.cc:4584] TS e49cba59f887407699d050cf8d8a86c6 (127.0.97.1:36893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 060b26f1de8249c69c88fe1818a8d1d2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
I20260929 11:24:33.872550 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.853945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.0.97.2:34017)
0929 11:24:33.854151 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:33.854157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:33.854437 (+ 280us) server_negotiation.cc:407] Connection header received
0929 11:24:33.854648 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:33.854654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:33.854759 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:33.854862 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:33.856725 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.857511 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.859465 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:33.860022 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:33.860125 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:33.866937 (+ 6812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:33.866953 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:33.866982 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:33.867040 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:33.869988 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.870670 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.870677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.870679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.870799 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:33.871102 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:33.871105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:33.871107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:33.871475 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:33.871698 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:33.872004 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 11:24:33.872167 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:24:33.873245 16765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:33.910075 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.909441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50490)
0929 11:24:33.909757 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.909885 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260929 11:24:33.911386 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.910917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50492)
0929 11:24:33.911124 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.911233 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 11:24:33.972029 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.971444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50506)
0929 11:24:33.971699 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.971821 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 11:24:33.973214 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:33.972799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50522)
0929 11:24:33.972982 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:33.973083 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 11:24:33.983795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 16642
I20260929 11:24:34.000764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 16773
I20260929 11:24:34.015411 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:34.001578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.0.97.3:41677)
0929 11:24:34.001762 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:34.001771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:34.001818 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:24:34.002111 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:34.002117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:34.002200 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:34.002307 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:34.004140 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:34.005429 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:34.007265 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:34.007642 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:34.007738 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:34.010225 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:34.010240 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:34.010258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:34.010307 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:34.013157 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:34.013783 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:34.013790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:34.013792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:34.013923 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:34.014220 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:34.014223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:34.014225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:34.014557 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:34.014773 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:34.015012 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 11:24:34.015162 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:24:34.016043 16896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:34.053754 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:34.053194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50526)
0929 11:24:34.053415 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:34.053524 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:24:34.054926 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:34.054486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50534)
0929 11:24:34.054709 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:34.054798 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 11:24:34.104899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 16773
I20260929 11:24:34.121755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 16419
I20260929 11:24:34.122943 16482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:24:34.240137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 16419
I20260929 11:24:34.240186 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:34.239461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36893 (local address 127.0.0.1:50538)
0929 11:24:34.239883 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:34.240000 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260929 11:24:54.261812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46683
--webserver_interface=127.0.97.62
--webserver_port=34383
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:24:54.589918 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:54.590209 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:54.590350 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:54.597715 16948 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:24:54.597817 16948 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:24:54.597909 16948 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:24:54.598037 16948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:24:54.598137 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:54.598271 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:54.598359 16948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:24:54.598445 16948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:24:54.608440 16948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:54.609766 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:54.611124 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:54.622740 16954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:54.622709 16953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:54.622771 16956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:54.623790 16948 server_base.cc:1034] running on GCE node
I20260929 11:24:54.624521 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:54.625806 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:54.627015 16948 hybrid_clock.cc:648] HybridClock initialized: now 1790681094626948 us; error 46 us; skew 500 ppm
I20260929 11:24:54.627329 16948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:54 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:54.630376 16948 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:24:54.631940 16948 webserver.cc:460] Webserver started at http://127.0.97.62:34383/ using document root <none> and password file <none>
I20260929 11:24:54.632674 16948 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:54.632792 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:54.637876 16948 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:24:54.640452 16963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:54.641710 16948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:24:54.641870 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "6b2427e3ba2a420ba3a205167b3c4921"
format_stamp: "Formatted at 2026-09-29 11:24:08 on dist-test-slave-srst"
server_key: "9b9643b59ba2f4c059bb85026cd34244"
server_key_iv: "39b90958b4f7e350c7c1b1a5b400c701"
server_key_version: "encryptionkey@0"
I20260929 11:24:54.642774 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:54.666297 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:54.670035 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:54.670341 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:54.688292 16948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46683
I20260929 11:24:54.688334 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46683 every 8 connection(s)
I20260929 11:24:54.689848 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:24:54.695387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 16948
I20260929 11:24:54.696403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36893
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46683
--webserver_port=35939
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:24:54.697204 17015 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:24:54.702314 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap starting.
I20260929 11:24:54.723137 17015 log.cc:824] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:54.738725 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:24:54.739248 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Bootstrap complete.
I20260929 11:24:54.748486 17015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:54.749451 17015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2427e3ba2a420ba3a205167b3c4921, State: Initialized, Role: FOLLOWER
I20260929 11:24:54.750108 17015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:54.750347 17015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:24:54.750483 17015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:24:54.750675 17015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:24:54.754429 17015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:54.754933 17015 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2427e3ba2a420ba3a205167b3c4921; no voters:
I20260929 11:24:54.755640 17015 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 11:24:54.755786 17019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:24:54.757011 17019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [term 3 LEADER]: Becoming Leader. State: Replica: 6b2427e3ba2a420ba3a205167b3c4921, State: Running, Role: LEADER
I20260929 11:24:54.757603 17019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } }
I20260929 11:24:54.757872 17015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:24:54.760527 17021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2427e3ba2a420ba3a205167b3c4921. Latest consensus state: current_term: 3 leader_uuid: "6b2427e3ba2a420ba3a205167b3c4921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } } }
I20260929 11:24:54.760563 17020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6b2427e3ba2a420ba3a205167b3c4921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2427e3ba2a420ba3a205167b3c4921" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46683 } } }
I20260929 11:24:54.760944 17021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: This master's current role is: LEADER
I20260929 11:24:54.761029 17020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921 [sys.catalog]: This master's current role is: LEADER
I20260929 11:24:54.770057 17026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:24:54.778983 17026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:24:54.780836 17026 catalog_manager.cc:1259] Loaded cluster ID: f9739324ad0646ef99071ae4aa46dcc1
I20260929 11:24:54.781002 17026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:24:54.786258 17026 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:24:54.788404 17026 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Loaded TSK: 0
I20260929 11:24:54.788621 17026 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b2427e3ba2a420ba3a205167b3c4921: Loaded TSK: 1
F20260929 11:24:54.789994 17026 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5f5bf779ad google::LogMessage::Fail() at ??:0
@ 0x7f5f5bf7a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5f5bf774de google::LogMessage::Flush() at ??:0
@ 0x7f5f5bf78a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5f5fcdb7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5f5fcca73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5f719a61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5f719a5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5f71a312a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5f71a310c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5f71a7b5e9 std::function<>::operator()() at ??:0
@ 0x7f5f719a9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5f719a7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5f71a2f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5f71a2f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5f5ebc3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5f5ebc691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5f5ebc6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5f5eba1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5f61e7b6db start_thread at ??:0
@ 0x7f5f5ae1161f clone at ??:0
W20260929 11:24:55.082875 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:55.083142 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:55.083302 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:55.090456 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:55.090559 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:55.090744 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:24:55.101025 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35939
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:55.102335 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:55.103667 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:55.117496 17062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:55.117623 17063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:55.117597 17065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:55.118615 17017 server_base.cc:1034] running on GCE node
I20260929 11:24:55.119251 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:55.120234 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:55.121431 17017 hybrid_clock.cc:648] HybridClock initialized: now 1790681095121385 us; error 49 us; skew 500 ppm
I20260929 11:24:55.121760 17017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:55 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:55.124910 17017 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:24:55.126441 17017 webserver.cc:460] Webserver started at http://127.0.97.1:35939/ using document root <none> and password file <none>
I20260929 11:24:55.127153 17017 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:55.127312 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:55.132973 17017 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:24:55.135787 17072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:55.137169 17017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:24:55.137354 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "e49cba59f887407699d050cf8d8a86c6"
format_stamp: "Formatted at 2026-09-29 11:24:09 on dist-test-slave-srst"
server_key: "5807c96a5c37f83294e4c79c5c569643"
server_key_iv: "3afb1f0352042f7699ddf27d890dbdf3"
server_key_version: "encryptionkey@0"
I20260929 11:24:55.138330 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:55.162882 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:55.167018 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:55.167322 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:55.168323 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:55.170220 17079 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 11:24:55.176041 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 11:24:55.176229 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 11:24:55.176380 17017 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 11:24:55.179522 17017 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 11:24:55.179627 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 11:24:55.180038 17079 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Bootstrap starting.
I20260929 11:24:55.210638 17079 log.cc:824] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:24:55.223658 17017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36893
I20260929 11:24:55.223688 17186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36893 every 8 connection(s)
I20260929 11:24:55.225363 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:24:55.228168 17079 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:24:55.228864 17079 tablet_bootstrap.cc:492] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Bootstrap complete.
I20260929 11:24:55.230384 17079 ts_tablet_manager.cc:1397] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Time spent bootstrapping tablet: real 0.051s user 0.036s sys 0.012s
I20260929 11:24:55.230679 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.228472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:53975)
0929 11:24:55.229452 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.229672 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":462,"threads_started":1}
W20260929 11:24:55.231496 17187 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46683 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46683: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
I20260929 11:24:55.232501 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.232042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:45037)
0929 11:24:55.232240 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.232321 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 11:24:55.233049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17017
I20260929 11:24:55.233620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34017
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46683
--webserver_port=36377
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:24:55.237674 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.237217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:35007)
0929 11:24:55.237403 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.237503 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 11:24:55.238023 17187 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:24:55.237315 17079 raft_consensus.cc:348] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:55.238554 17079 raft_consensus.cc:729] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cba59f887407699d050cf8d8a86c6, State: Initialized, Role: FOLLOWER
I20260929 11:24:55.239226 17079 consensus_queue.cc:260] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:55.240875 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.240512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:54635)
0929 11:24:55.240682 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.240755 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:24:55.240882 17079 ts_tablet_manager.cc:1428] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6: Time spent starting tablet: real 0.010s user 0.010s sys 0.002s
I20260929 11:24:55.241916 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.241581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:45841)
0929 11:24:55.241719 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.241783 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:24:55.560526 17191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:55.560854 17191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:55.560948 17191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:55.568042 17191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:55.568147 17191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:55.568260 17191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:24:55.577955 17191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36377
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:55.579156 17191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:55.580399 17191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:55.593238 17197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:55.593276 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:55.593299 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:55.594390 17191 server_base.cc:1034] running on GCE node
I20260929 11:24:55.594997 17191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:55.595896 17191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:55.597124 17191 hybrid_clock.cc:648] HybridClock initialized: now 1790681095597076 us; error 43 us; skew 500 ppm
I20260929 11:24:55.597463 17191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:55 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:55.600409 17191 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:24:55.601955 17191 webserver.cc:460] Webserver started at http://127.0.97.2:36377/ using document root <none> and password file <none>
I20260929 11:24:55.602595 17191 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:55.602735 17191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:55.607942 17191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 11:24:55.610723 17206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:55.612066 17191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:24:55.612244 17191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "a2771a73d7d24637929594cab380594e"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "8293c16653498d57d10e2e0bffb3b9e3"
server_key_iv: "3d9956c95ed8550f7d0565c2075c9847"
server_key_version: "encryptionkey@0"
I20260929 11:24:55.613296 17191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:55.642624 17191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:55.646983 17191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:55.647300 17191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:55.648315 17191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:55.649777 17191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:55.649912 17191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:55.650060 17191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:55.650141 17191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:55.682644 17191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34017
I20260929 11:24:55.682732 17318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34017 every 8 connection(s)
I20260929 11:24:55.684069 17191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:24:55.687801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17191
I20260929 11:24:55.688385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41677
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46683
--webserver_port=43533
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:24:55.689263 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.687174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:37783)
0929 11:24:55.688053 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.688300 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":400,"threads_started":1}
W20260929 11:24:55.689893 17319 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46683 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46683: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
I20260929 11:24:55.690733 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.690438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:42593)
0929 11:24:55.690573 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.690634 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 11:24:55.691619 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:55.691329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:36761)
0929 11:24:55.691449 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:55.691518 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:24:55.691920 17319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:24:56.008059 17322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:24:56.008325 17322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:24:56.008419 17322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:24:56.015655 17322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:24:56.015762 17322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:24:56.015873 17322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:24:56.025563 17322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43533
--tserver_master_addrs=127.0.97.62:46683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:24:56.026733 17322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:24:56.027981 17322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:24:56.041203 17327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:56.041268 17328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:24:56.041168 17330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:24:56.042429 17322 server_base.cc:1034] running on GCE node
I20260929 11:24:56.042968 17322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:24:56.043933 17322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:24:56.045127 17322 hybrid_clock.cc:648] HybridClock initialized: now 1790681096045074 us; error 45 us; skew 500 ppm
I20260929 11:24:56.045434 17322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:24:56 dist-test-slave-srst krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:24:56.048357 17322 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:24:56.049901 17322 webserver.cc:460] Webserver started at http://127.0.97.3:43533/ using document root <none> and password file <none>
I20260929 11:24:56.050611 17322 fs_manager.cc:359] Metadata directory not provided
I20260929 11:24:56.050758 17322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:24:56.055917 17322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:24:56.058804 17337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:56.060119 17322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:24:56.060319 17322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "9253461dbdf54068ad7f95be33a5ef5a"
format_stamp: "Formatted at 2026-09-29 11:24:10 on dist-test-slave-srst"
server_key: "5f0c422a62be30390a15433a886f9281"
server_key_iv: "b54da10fabc20242cf12588bcb198c49"
server_key_version: "encryptionkey@0"
I20260929 11:24:56.061405 17322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:24:56.082086 17322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:24:56.086563 17322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:24:56.086844 17322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:24:56.087852 17322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:24:56.089490 17322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:24:56.089628 17322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:56.089758 17322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:24:56.089844 17322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:24:56.123042 17322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41677
I20260929 11:24:56.123129 17449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41677 every 8 connection(s)
I20260929 11:24:56.124547 17322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:24:56.129599 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.127475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:46199)
0929 11:24:56.128432 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.128736 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":272,"threads_started":1}
W20260929 11:24:56.130240 17450 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46683 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46683: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
I20260929 11:24:56.131166 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.130867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:37829)
0929 11:24:56.130998 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.131072 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:24:56.131948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17322
I20260929 11:24:56.132200 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.131736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:42845)
0929 11:24:56.131933 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.132063 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 11:24:56.132507 17450 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:24:56.243549 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.242707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:37473)
0929 11:24:56.243157 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.243273 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260929 11:24:56.693262 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.692459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:45079)
0929 11:24:56.692949 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.693050 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":281,"threads_started":1}
I20260929 11:24:56.700749 17456 raft_consensus.cc:482] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:24:56.701011 17456 raft_consensus.cc:504] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:56.702625 17456 leader_election.cc:288] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
Sep 29 11:24:56 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 29 11:24:56 dist-test-slave-srst krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:24:56.725428 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.703544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36083 (local address 127.0.97.2:34017)
0929 11:24:56.704981 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:56.704993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:56.705030 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:24:56.706804 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:56.706812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:56.707223 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:56.707577 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:56.709605 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.710776 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.712454 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.713127 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.713326 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:56.717757 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:56.717766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:56.717783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:56.717833 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:56.721688 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:56.722549 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:56.722557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:56.722564 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:56.722734 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:56.723223 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:56.723228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:56.723237 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:56.723701 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:56.723917 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:56.724688 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 11:24:56.725062 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247,"thread_start_us":1088,"threads_started":1}
I20260929 11:24:56.725450 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.703040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34017 (local address 127.0.97.1:36083)
0929 11:24:56.704707 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.704727 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:56.704837 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:56.707719 (+ 2882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:56.707728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:56.708525 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:56.709353 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:56.709385 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.710941 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.710950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:56.712315 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:56.712327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.713384 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.713392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:56.713617 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:56.714478 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:56.714507 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:56.717504 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:56.721840 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:56.721847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:56.721858 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:56.722410 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:56.722882 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:56.722885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:56.722887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:56.723083 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:56.723867 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:56.723872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:56.724183 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 11:24:56.724441 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 11:24:56.725011 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":259,"spinlock_wait_cycles":25216,"thread_start_us":1303,"threads_started":1}
I20260929 11:24:56.725445 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.703149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59429 (local address 127.0.97.3:41677)
0929 11:24:56.705254 (+ 2105us) server_negotiation.cc:206] Beginning negotiation
0929 11:24:56.705265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:24:56.705309 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:24:56.706872 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:24:56.706881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:24:56.707249 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:24:56.707576 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:24:56.709589 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.710743 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.712446 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.713124 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.713319 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:56.717783 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:24:56.717791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:24:56.717802 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:24:56.717843 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:24:56.721697 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:56.722552 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:56.722560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:56.722565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:56.722734 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:24:56.723186 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:24:56.723192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:24:56.723196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:24:56.723695 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:24:56.723901 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 11:24:56.724691 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 11:24:56.725057 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":1834,"threads_started":1}
I20260929 11:24:56.725528 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:56.702874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41677 (local address 127.0.97.1:59429)
0929 11:24:56.703089 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:56.703126 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:24:56.703776 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:24:56.707718 (+ 3942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:24:56.707737 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:24:56.708529 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:24:56.709353 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:56.709385 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.710929 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.710933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:56.712307 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:24:56.712318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:24:56.713341 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:24:56.713348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:24:56.713616 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:24:56.714556 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:24:56.714572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:24:56.717481 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:24:56.721910 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:56.721923 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:56.721926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:56.722412 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:56.722861 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:24:56.722867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:24:56.722871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:24:56.723075 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:24:56.723861 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:24:56.723868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:24:56.724183 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 11:24:56.724445 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 11:24:56.725147 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":434,"spinlock_wait_cycles":29056}
W20260929 11:24:56.728569 17076 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
W20260929 11:24:56.728868 17074 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2771a73d7d24637929594cab380594e (127.0.97.2:34017): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:24:56.728986 17074 leader_election.cc:302] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49cba59f887407699d050cf8d8a86c6; no voters: 9253461dbdf54068ad7f95be33a5ef5a, a2771a73d7d24637929594cab380594e
I20260929 11:24:56.729326 17456 raft_consensus.cc:2726] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:24:57.133917 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:57.133011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:37017)
0929 11:24:57.133460 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:57.133599 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":211,"threads_started":1}
I20260929 11:24:57.245162 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:57.244375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:59145)
0929 11:24:57.244889 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:57.244974 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":319,"threads_started":1}
I20260929 11:24:57.694931 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:57.694094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:38113)
0929 11:24:57.694673 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:57.694763 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":331,"threads_started":1}
I20260929 11:24:58.135471 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:58.134821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:59565)
0929 11:24:58.135249 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:58.135327 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260929 11:24:58.246798 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:58.246016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:42021)
0929 11:24:58.246516 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:58.246601 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":220,"threads_started":1}
I20260929 11:24:58.696516 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:58.695818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:56813)
0929 11:24:58.696258 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:58.696351 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260929 11:24:58.760006 17466 raft_consensus.cc:482] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:24:58.760186 17466 raft_consensus.cc:504] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:24:58.760887 17466 leader_election.cc:288] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
W20260929 11:24:58.761847 17076 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
W20260929 11:24:58.762058 17074 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2771a73d7d24637929594cab380594e (127.0.97.2:34017): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:24:58.762195 17074 leader_election.cc:302] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49cba59f887407699d050cf8d8a86c6; no voters: 9253461dbdf54068ad7f95be33a5ef5a, a2771a73d7d24637929594cab380594e
I20260929 11:24:58.762470 17466 raft_consensus.cc:2726] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:24:59.137010 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:59.136295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:42615)
0929 11:24:59.136785 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:59.136861 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":219,"threads_started":1}
I20260929 11:24:59.248540 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:59.247657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:45177)
0929 11:24:59.248201 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:59.248304 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":268,"threads_started":1}
I20260929 11:24:59.698125 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:24:59.697453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:41213)
0929 11:24:59.697868 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 11:24:59.697946 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":216,"threads_started":1}
I20260929 11:25:00.138481 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:00.137829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:45537)
0929 11:25:00.138222 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:00.138305 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":194,"threads_started":1}
I20260929 11:25:00.250394 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:00.249694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:57295)
0929 11:25:00.250128 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:00.250207 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260929 11:25:00.699787 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:00.699016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:42827)
0929 11:25:00.699550 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:00.699635 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":331,"threads_started":1}
I20260929 11:25:01.139992 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:01.139356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:56763)
0929 11:25:01.139774 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:01.139845 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":245,"threads_started":1}
I20260929 11:25:01.251921 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:01.251273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:51383)
0929 11:25:01.251676 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:01.251754 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":215,"threads_started":1}
I20260929 11:25:01.701388 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:01.700641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:46431)
0929 11:25:01.701123 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:01.701201 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260929 11:25:02.063934 17476 raft_consensus.cc:482] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:02.064198 17476 raft_consensus.cc:504] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:25:02.064991 17476 leader_election.cc:288] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
W20260929 11:25:02.065999 17074 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2771a73d7d24637929594cab380594e (127.0.97.2:34017): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
W20260929 11:25:02.066185 17076 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:25:02.066282 17076 leader_election.cc:302] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49cba59f887407699d050cf8d8a86c6; no voters: 9253461dbdf54068ad7f95be33a5ef5a, a2771a73d7d24637929594cab380594e
I20260929 11:25:02.066505 17476 raft_consensus.cc:2726] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:25:02.141633 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:02.140943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:59747)
0929 11:25:02.141378 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:02.141458 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260929 11:25:02.253732 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:02.252943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:53069)
0929 11:25:02.253433 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:02.253529 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260929 11:25:02.702979 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:02.702227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:38049)
0929 11:25:02.702715 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:02.702790 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":227,"threads_started":1}
I20260929 11:25:03.143250 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:03.142434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:42519)
0929 11:25:03.142948 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:03.143073 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":219,"threads_started":1}
I20260929 11:25:03.255273 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:03.254579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:41015)
0929 11:25:03.255033 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:03.255110 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":251,"threads_started":1}
I20260929 11:25:03.704543 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:03.703884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:52149)
0929 11:25:03.704296 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:03.704382 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":228,"threads_started":1}
I20260929 11:25:04.144899 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:04.144168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:45191)
0929 11:25:04.144636 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:04.144714 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260929 11:25:04.256963 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:04.256224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:58855)
0929 11:25:04.256697 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:04.256776 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260929 11:25:04.706221 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:04.705525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:46737)
0929 11:25:04.705982 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:04.706064 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":219,"threads_started":1}
I20260929 11:25:05.146426 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:05.145758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:50049)
0929 11:25:05.146210 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:05.146282 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260929 11:25:05.258597 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:05.257931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:43903)
0929 11:25:05.258341 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:05.258434 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260929 11:25:05.708135 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:05.707286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:58249)
0929 11:25:05.707825 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:05.707907 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":310,"threads_started":1}
I20260929 11:25:06.148183 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:06.147391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:55701)
0929 11:25:06.147912 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:06.147988 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":308,"threads_started":1}
I20260929 11:25:06.260193 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:06.259504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:51295)
0929 11:25:06.259944 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:06.260021 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260929 11:25:06.709745 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:06.709036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:35759)
0929 11:25:06.709494 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:06.709573 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260929 11:25:06.971014 17492 raft_consensus.cc:482] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:06.971215 17492 raft_consensus.cc:504] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9253461dbdf54068ad7f95be33a5ef5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41677 } } peers { permanent_uuid: "a2771a73d7d24637929594cab380594e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34017 } } peers { permanent_uuid: "e49cba59f887407699d050cf8d8a86c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36893 } }
I20260929 11:25:06.971858 17492 leader_election.cc:288] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677), a2771a73d7d24637929594cab380594e (127.0.97.2:34017)
W20260929 11:25:06.972812 17076 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9253461dbdf54068ad7f95be33a5ef5a (127.0.97.3:41677): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
W20260929 11:25:06.973001 17074 leader_election.cc:341] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2771a73d7d24637929594cab380594e (127.0.97.2:34017): Not found: Tablet not found: 060b26f1de8249c69c88fe1818a8d1d2
I20260929 11:25:06.973187 17074 leader_election.cc:302] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49cba59f887407699d050cf8d8a86c6; no voters: 9253461dbdf54068ad7f95be33a5ef5a, a2771a73d7d24637929594cab380594e
I20260929 11:25:06.973436 17492 raft_consensus.cc:2726] T 060b26f1de8249c69c88fe1818a8d1d2 P e49cba59f887407699d050cf8d8a86c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:25:07.149729 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:07.149037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:47013)
0929 11:25:07.149491 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:07.149574 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260929 11:25:07.261917 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:07.261094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:54145)
0929 11:25:07.261594 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:07.261672 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":236,"threads_started":1}
I20260929 11:25:07.711330 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:07.710645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:51015)
0929 11:25:07.711090 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:07.711164 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":221,"threads_started":1}
I20260929 11:25:08.151544 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:08.150687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:51365)
0929 11:25:08.151214 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:08.151304 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260929 11:25:08.263626 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:08.262964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:49127)
0929 11:25:08.263385 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:08.263457 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":207,"threads_started":1}
I20260929 11:25:08.712957 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:08.712212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:53757)
0929 11:25:08.712722 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:08.712798 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260929 11:25:09.153173 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:09.152363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:51725)
0929 11:25:09.152941 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:09.153013 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":371,"threads_started":1}
I20260929 11:25:09.265554 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:09.264656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:36975)
0929 11:25:09.265277 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:09.265356 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":352,"threads_started":1}
I20260929 11:25:09.714574 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:09.713834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:44571)
0929 11:25:09.714290 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:09.714364 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":252,"threads_started":1}
I20260929 11:25:10.154831 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:10.154055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:46339)
0929 11:25:10.154535 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:10.154619 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
I20260929 11:25:10.267500 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:10.266736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.1:33587)
0929 11:25:10.267194 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:10.267287 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260929 11:25:10.716708 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:10.715808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.2:54147)
0929 11:25:10.716410 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:10.716498 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":320,"threads_started":1}
I20260929 11:25:11.135077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17017
I20260929 11:25:11.154682 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:11.136501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.97.1:36893)
0929 11:25:11.137126 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:11.137134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:11.137221 (+ 87us) server_negotiation.cc:407] Connection header received
0929 11:25:11.137382 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:11.137389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:11.137611 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:11.137750 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:11.139144 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:11.140159 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:11.141444 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:11.142057 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:11.142166 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:11.148901 (+ 6735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:11.148915 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:11.148931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:11.148969 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:11.151686 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:11.152450 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:11.152457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:11.152460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:11.152654 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:11.153046 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:11.153051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:11.153054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:11.153429 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:11.153767 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:11.154045 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 11:25:11.154346 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":320,"threads_started":1}
I20260929 11:25:11.156054 17182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:11.156740 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:11.155949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46683 (local address 127.0.97.3:50549)
0929 11:25:11.156452 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:11.156544 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":284,"threads_started":1}
I20260929 11:25:11.248754 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17017
I20260929 11:25:11.266618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17191
I20260929 11:25:11.279310 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:11.267336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.0.97.2:34017)
0929 11:25:11.267831 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:11.267835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:11.267852 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:25:11.267908 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:11.267911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:11.267972 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:11.268064 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:11.269450 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:11.270236 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:11.271333 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:11.271744 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:11.271834 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:11.274317 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:11.274321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:11.274324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:11.274356 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:11.277202 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:11.277762 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:11.277766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:11.277768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:11.277879 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:11.278171 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:11.278174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:11.278176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:11.278427 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:11.278627 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:11.278865 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 11:25:11.279034 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":289,"threads_started":1}
I20260929 11:25:11.280007 17314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:11.366395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17191
I20260929 11:25:11.383410 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17322
I20260929 11:25:11.397918 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:11.384324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.97.3:41677)
0929 11:25:11.384950 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:11.384955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:11.384973 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:25:11.385031 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:11.385035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:11.385118 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:11.385209 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:11.386750 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:11.387590 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:11.389259 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:11.389685 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:11.389794 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:11.392578 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:11.392584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:11.392653 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:11.392778 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:11.395372 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:11.396059 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:11.396063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:11.396065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:11.396208 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:11.396531 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:11.396534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:11.396536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:11.396950 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:11.397179 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:11.397429 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 11:25:11.397584 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":339,"threads_started":1}
I20260929 11:25:11.398612 17445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:11.485961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17322
2026-09-29T11:25:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64249 ms)
[----------] 30 tests from SecurityITest (289587 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:11 dist-test-slave-srst krb5kdc[17514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:11 dist-test-slave-srst krb5kdc[17514](info): set up 2 sockets
Sep 29 11:25:11 dist-test-slave-srst krb5kdc[17514](info): commencing operation
krb5kdc: starting...
W20260929 11:25:12.588635 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:12 dist-test-slave-srst krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:25:12.820394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38611
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:13.149911 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:13.150213 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:13.150357 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:13.157473 17527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:13.157605 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:13.157716 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:13.157827 17527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:13.157912 17527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:13.167217 17527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:13.168448 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:13.169718 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:13.181671 17532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:13.181609 17533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:13.181567 17535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:13.182139 17527 server_base.cc:1034] running on GCE node
I20260929 11:25:13.182984 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:13.184450 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:13.185688 17527 hybrid_clock.cc:648] HybridClock initialized: now 1790681113185642 us; error 34 us; skew 500 ppm
I20260929 11:25:13.186060 17527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:13 dist-test-slave-srst krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:13.189108 17527 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:25:13.190591 17527 webserver.cc:460] Webserver started at http://127.0.97.62:36369/ using document root <none> and password file <none>
I20260929 11:25:13.191221 17527 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:13.191318 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:13.191581 17527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:13.194490 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "d393cbb09af1449b9455caf61e61ec7e"
format_stamp: "Formatted at 2026-09-29 11:25:13 on dist-test-slave-srst"
server_key: "3a07d6b6b7bdda08438444a494863d88"
server_key_iv: "c23dcd2607928ef976c768562ee0e835"
server_key_version: "encryptionkey@0"
I20260929 11:25:13.195174 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "d393cbb09af1449b9455caf61e61ec7e"
format_stamp: "Formatted at 2026-09-29 11:25:13 on dist-test-slave-srst"
server_key: "3a07d6b6b7bdda08438444a494863d88"
server_key_iv: "c23dcd2607928ef976c768562ee0e835"
server_key_version: "encryptionkey@0"
I20260929 11:25:13.200148 17527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:13.203629 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:13.204985 17527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:13.205163 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "d393cbb09af1449b9455caf61e61ec7e"
format_stamp: "Formatted at 2026-09-29 11:25:13 on dist-test-slave-srst"
server_key: "3a07d6b6b7bdda08438444a494863d88"
server_key_iv: "c23dcd2607928ef976c768562ee0e835"
server_key_version: "encryptionkey@0"
I20260929 11:25:13.205364 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:13.217168 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:13.221517 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:13.221797 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:13.240206 17527 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38611
I20260929 11:25:13.240295 17593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38611 every 8 connection(s)
I20260929 11:25:13.241626 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:13.245373 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:13.251556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17527
I20260929 11:25:13.251750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:13.252539 388 external_mini_cluster.cc:1428] Setting key 102dfc9c9d97f02269ae6e8ebeac17a2
I20260929 11:25:13.256660 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e: Bootstrap starting.
I20260929 11:25:13.260550 17594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:13.261739 17594 log.cc:824] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:13.264938 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e: No bootstrap required, opened a new log
Sep 29 11:25:13 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:13.273695 17594 raft_consensus.cc:348] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d393cbb09af1449b9455caf61e61ec7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38611 } }
I20260929 11:25:13.274101 17594 raft_consensus.cc:374] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:13.274225 17594 raft_consensus.cc:729] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d393cbb09af1449b9455caf61e61ec7e, State: Initialized, Role: FOLLOWER
I20260929 11:25:13.274926 17594 consensus_queue.cc:260] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d393cbb09af1449b9455caf61e61ec7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38611 } }
I20260929 11:25:13.275249 17594 raft_consensus.cc:388] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:13.275420 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:13.275633 17594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:13.276300 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:13.254622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.0.97.62:38611)
0929 11:25:13.255450 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:13.255463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:13.255505 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:25:13.257351 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:13.257374 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:13.257843 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:13.258174 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:13.260115 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:13.261574 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:13.263278 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:13.263986 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:13.264127 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:13.268869 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:13.268877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:13.268896 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:13.268941 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:13.271937 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:13.272748 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:13.272755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:13.272763 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:13.272939 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:13.273322 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:13.273327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:13.273336 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:13.273851 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:13.274172 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:13.274596 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 11:25:13.275076 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":398,"threads_started":1}
I20260929 11:25:13.278595 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d393cbb09af1449b9455caf61e61ec7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38611 } }
I20260929 11:25:13.279171 17594 leader_election.cc:302] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d393cbb09af1449b9455caf61e61ec7e; no voters:
I20260929 11:25:13.279776 17594 leader_election.cc:288] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:13.279928 17599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:13.280959 17599 raft_consensus.cc:686] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [term 1 LEADER]: Becoming Leader. State: Replica: d393cbb09af1449b9455caf61e61ec7e, State: Running, Role: LEADER
I20260929 11:25:13.281559 17599 consensus_queue.cc:237] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d393cbb09af1449b9455caf61e61ec7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38611 } }
I20260929 11:25:13.282040 17594 sys_catalog.cc:564] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:13.284538 17600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d393cbb09af1449b9455caf61e61ec7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d393cbb09af1449b9455caf61e61ec7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38611 } } }
I20260929 11:25:13.284613 17601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d393cbb09af1449b9455caf61e61ec7e. Latest consensus state: current_term: 1 leader_uuid: "d393cbb09af1449b9455caf61e61ec7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d393cbb09af1449b9455caf61e61ec7e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38611 } } }
I20260929 11:25:13.284981 17601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:13.284977 17600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:13.286602 17606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:13.292827 17606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:13.302112 17606 catalog_manager.cc:1347] Generated new cluster ID: 7a0b2b76950f4d79a85057835d827aff
I20260929 11:25:13.302241 17606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:13.311627 17606 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:13.313126 17606 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:13.324322 17606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e: Generated new TSK 0
I20260929 11:25:13.325496 17606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:25:13.393075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38611
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:13.722604 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:13.722890 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:13.723050 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:13.730574 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:13.730716 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:13.730880 17620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:25:13.740746 17620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:13.742053 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:13.743415 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:13.756776 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:13.756836 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:13.756868 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:13.757716 17620 server_base.cc:1034] running on GCE node
I20260929 11:25:13.758378 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:13.759349 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:13.760548 17620 hybrid_clock.cc:648] HybridClock initialized: now 1790681113760503 us; error 49 us; skew 500 ppm
I20260929 11:25:13.760912 17620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:13 dist-test-slave-srst krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:13.764076 17620 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:25:13.765612 17620 webserver.cc:460] Webserver started at http://127.0.97.1:35609/ using document root <none> and password file <none>
I20260929 11:25:13.766328 17620 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:13.766475 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:13.766788 17620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:13.769733 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "1284ed6811d04a6a8242e0e0713a47ab"
format_stamp: "Formatted at 2026-09-29 11:25:13 on dist-test-slave-srst"
server_key: "f9c327d5ad12d8bb3914cdc55d99e0a7"
server_key_iv: "fccdb09a88d04d0d6df71d3a8d4400e4"
server_key_version: "encryptionkey@0"
I20260929 11:25:13.770563 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1284ed6811d04a6a8242e0e0713a47ab"
format_stamp: "Formatted at 2026-09-29 11:25:13 on dist-test-slave-srst"
server_key: "f9c327d5ad12d8bb3914cdc55d99e0a7"
server_key_iv: "fccdb09a88d04d0d6df71d3a8d4400e4"
server_key_version: "encryptionkey@0"
I20260929 11:25:13.776021 17620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:13.779606 17635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:13.780994 17620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:13.781168 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "1284ed6811d04a6a8242e0e0713a47ab"
format_stamp: "Formatted at 2026-09-29 11:25:13 on dist-test-slave-srst"
server_key: "f9c327d5ad12d8bb3914cdc55d99e0a7"
server_key_iv: "fccdb09a88d04d0d6df71d3a8d4400e4"
server_key_version: "encryptionkey@0"
I20260929 11:25:13.781385 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:13.802886 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:13.807101 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:13.807413 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:13.808527 17620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:13.810074 17620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:13.810150 17620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:13.810316 17620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:13.810377 17620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:13.841742 17620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33393
I20260929 11:25:13.841820 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33393 every 8 connection(s)
I20260929 11:25:13.843020 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:25:13.846717 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17620
I20260929 11:25:13.846956 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:25:13.847882 388 external_mini_cluster.cc:1428] Setting key d3e90dff8738f291133ee7ef77b3ca8d
Sep 29 11:25:13 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:13.865257 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:13.845725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33005 (local address 127.0.97.62:38611)
0929 11:25:13.846174 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:13.846179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:13.847473 (+ 1294us) server_negotiation.cc:407] Connection header received
0929 11:25:13.849666 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:13.849672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:13.849759 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:13.849904 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:13.851403 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:13.852115 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:13.853748 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:13.854448 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:13.854636 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:13.858264 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:13.858273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:13.858281 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:13.858323 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:13.861886 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:13.862625 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:13.862634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:13.862640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:13.862786 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:13.863217 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:13.863225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:13.863231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:13.863601 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:13.863755 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:13.864404 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 11:25:13.864729 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260929 11:25:13.865875 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:13.846347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38611 (local address 127.0.97.1:33005)
0929 11:25:13.847282 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:13.847334 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:13.849400 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:13.850145 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:13.850153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:13.850668 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:13.851237 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:13.851254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:13.852263 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:13.852268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:13.853534 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:13.853546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:13.854698 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:13.854705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:13.854965 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:13.855634 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:13.855665 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:13.858044 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:13.862030 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:13.862038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:13.862051 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:13.862494 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:13.862892 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:13.862895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:13.862897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:13.863081 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:13.863717 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:13.863722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:13.864020 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 11:25:13.864241 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 11:25:13.864702 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":1466,"threads_started":1}
I20260929 11:25:13.867437 17748 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38611
I20260929 11:25:13.867743 17748 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:13.868453 17748 heartbeater.cc:511] Master 127.0.97.62:38611 requested a full tablet report, sending...
I20260929 11:25:13.870587 17559 ts_manager.cc:194] Registered new tserver with Master: 1284ed6811d04a6a8242e0e0713a47ab (127.0.97.1:33393)
I20260929 11:25:13.872052 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33005
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:25:13.904251 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38611
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:14.243963 17754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:14.244256 17754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:14.244415 17754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:14.251403 17754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:14.251543 17754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:14.251741 17754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:25:14.261395 17754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:14.262681 17754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:14.264000 17754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:14.276940 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:14.276834 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:14.276865 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:14.277941 17754 server_base.cc:1034] running on GCE node
I20260929 11:25:14.278476 17754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:14.279408 17754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:14.280678 17754 hybrid_clock.cc:648] HybridClock initialized: now 1790681114280643 us; error 48 us; skew 500 ppm
I20260929 11:25:14.280997 17754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:14 dist-test-slave-srst krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:14.284060 17754 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:25:14.285614 17754 webserver.cc:460] Webserver started at http://127.0.97.2:37359/ using document root <none> and password file <none>
I20260929 11:25:14.286270 17754 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:14.286366 17754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:14.286614 17754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:14.289426 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "325b2588397a43b69f45a6d2ae13c230"
format_stamp: "Formatted at 2026-09-29 11:25:14 on dist-test-slave-srst"
server_key: "6eec38ec9ffd9872b745b0ce9d6e6b6d"
server_key_iv: "faf1456ea69617e431d03d83832fe1c6"
server_key_version: "encryptionkey@0"
I20260929 11:25:14.290158 17754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "325b2588397a43b69f45a6d2ae13c230"
format_stamp: "Formatted at 2026-09-29 11:25:14 on dist-test-slave-srst"
server_key: "6eec38ec9ffd9872b745b0ce9d6e6b6d"
server_key_iv: "faf1456ea69617e431d03d83832fe1c6"
server_key_version: "encryptionkey@0"
I20260929 11:25:14.295035 17754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:25:14.298298 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:14.299564 17754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:25:14.299690 17754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "325b2588397a43b69f45a6d2ae13c230"
format_stamp: "Formatted at 2026-09-29 11:25:14 on dist-test-slave-srst"
server_key: "6eec38ec9ffd9872b745b0ce9d6e6b6d"
server_key_iv: "faf1456ea69617e431d03d83832fe1c6"
server_key_version: "encryptionkey@0"
I20260929 11:25:14.299840 17754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:14.325924 17754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:14.330989 17754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:14.331354 17754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:14.332428 17754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:14.333736 17754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:14.333807 17754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:14.333961 17754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:14.334012 17754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:14.367416 17754 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44553
I20260929 11:25:14.367501 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44553 every 8 connection(s)
I20260929 11:25:14.368821 17754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:25:14.376384 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17754
I20260929 11:25:14.376567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:25:14.377152 388 external_mini_cluster.cc:1428] Setting key 44c612c6b5d7b2589d6f9ae4b7444147
Sep 29 11:25:14 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:14.390273 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.371325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53397 (local address 127.0.97.62:38611)
0929 11:25:14.371802 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:14.371808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:14.372715 (+ 907us) server_negotiation.cc:407] Connection header received
0929 11:25:14.374504 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:14.374509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:14.374575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:14.374695 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:14.376203 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.377355 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.379124 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.379776 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.379913 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.383331 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:14.383340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:14.383347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:14.383391 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:14.386730 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:14.387554 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:14.387561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:14.387565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:14.387733 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:14.388256 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:14.388265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:14.388271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:14.388658 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:14.388804 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:14.389490 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 11:25:14.389753 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":218,"threads_started":1}
I20260929 11:25:14.390827 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.371806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38611 (local address 127.0.97.2:53397)
0929 11:25:14.372508 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:14.372550 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:14.374237 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:14.374869 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:14.374887 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:14.375473 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:14.376014 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.376031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.377490 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.377494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.378976 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.378997 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.379974 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.379979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.380161 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.380863 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:14.380887 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:14.383111 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:14.386866 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:14.386902 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:14.386920 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:14.387439 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:14.387866 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:14.387870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:14.387872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:14.388110 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:14.388803 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:14.388809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:14.389094 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 11:25:14.389337 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 11:25:14.389704 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":301,"threads_started":1}
I20260929 11:25:14.392309 17882 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38611
I20260929 11:25:14.392648 17882 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:14.393486 17882 heartbeater.cc:511] Master 127.0.97.62:38611 requested a full tablet report, sending...
I20260929 11:25:14.395051 17559 ts_manager.cc:194] Registered new tserver with Master: 325b2588397a43b69f45a6d2ae13c230 (127.0.97.2:44553)
I20260929 11:25:14.395872 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53397
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:25:14.431946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38611
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:14.749186 17888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:14.749491 17888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:14.749670 17888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:14.756812 17888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:14.756950 17888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:14.757117 17888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:25:14.766472 17888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:14.767706 17888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:14.769068 17888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:14.781570 17894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:14.781692 17893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:14.782094 17896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:14.783128 17888 server_base.cc:1034] running on GCE node
I20260929 11:25:14.783716 17888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:14.784713 17888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:14.785940 17888 hybrid_clock.cc:648] HybridClock initialized: now 1790681114785848 us; error 79 us; skew 500 ppm
I20260929 11:25:14.786311 17888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:14 dist-test-slave-srst krb5kdc[17514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:14.789655 17888 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:25:14.791041 17888 webserver.cc:460] Webserver started at http://127.0.97.3:35793/ using document root <none> and password file <none>
I20260929 11:25:14.791692 17888 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:14.791791 17888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:14.792078 17888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:14.794922 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "65379b598b0c4da5a60a8bc8955478bc"
format_stamp: "Formatted at 2026-09-29 11:25:14 on dist-test-slave-srst"
server_key: "63e0714784f08b17b465141202927d87"
server_key_iv: "90f462245eaad5e6df0996f9a9476276"
server_key_version: "encryptionkey@0"
I20260929 11:25:14.795642 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "65379b598b0c4da5a60a8bc8955478bc"
format_stamp: "Formatted at 2026-09-29 11:25:14 on dist-test-slave-srst"
server_key: "63e0714784f08b17b465141202927d87"
server_key_iv: "90f462245eaad5e6df0996f9a9476276"
server_key_version: "encryptionkey@0"
I20260929 11:25:14.800498 17888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:25:14.803893 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:14.805208 17888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:14.805377 17888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "65379b598b0c4da5a60a8bc8955478bc"
format_stamp: "Formatted at 2026-09-29 11:25:14 on dist-test-slave-srst"
server_key: "63e0714784f08b17b465141202927d87"
server_key_iv: "90f462245eaad5e6df0996f9a9476276"
server_key_version: "encryptionkey@0"
I20260929 11:25:14.805567 17888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:14.820385 17888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:14.824465 17888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:14.824801 17888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:14.825711 17888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:14.826958 17888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:14.827064 17888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:14.827199 17888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:14.827268 17888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:14.858655 17888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34549
I20260929 11:25:14.858687 18015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34549 every 8 connection(s)
I20260929 11:25:14.859849 17888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:25:14.862138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 17888
I20260929 11:25:14.862327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:25:14.862965 388 external_mini_cluster.cc:1428] Setting key 49ca5b6daedaa13d9e4f3e3828b857ad
Sep 29 11:25:14 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:14.875002 17748 heartbeater.cc:503] Master 127.0.97.62:38611 was elected leader, sending a full tablet report...
I20260929 11:25:14.880153 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.862476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49513 (local address 127.0.97.62:38611)
0929 11:25:14.862639 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:14.862643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:14.864124 (+ 1481us) server_negotiation.cc:407] Connection header received
0929 11:25:14.866613 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:14.866617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:14.866683 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:14.866761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:14.868158 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.868954 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.870447 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.870864 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.870956 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.874110 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:14.874115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:14.874118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:14.874148 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:14.877300 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:14.878030 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:14.878037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:14.878041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:14.878165 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:14.878570 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:14.878585 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:14.878588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:14.878843 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:14.878998 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:14.879633 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 11:25:14.879850 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:25:14.881044 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.863079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38611 (local address 127.0.97.3:49513)
0929 11:25:14.863962 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:14.864002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:14.866374 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:14.866980 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:14.866988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:14.867491 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:14.868023 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.868040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.869090 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.869093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.870281 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.870293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.871058 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.871063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.871237 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.871818 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:14.871837 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:14.873932 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:14.877399 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:14.877407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:14.877420 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:14.877920 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:14.878272 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:14.878277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:14.878280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:14.878460 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:14.878974 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:14.878981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:14.879281 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 11:25:14.879527 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 11:25:14.879988 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":369,"threads_started":1}
I20260929 11:25:14.882299 18016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38611
I20260929 11:25:14.882587 18016 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:14.883397 18016 heartbeater.cc:511] Master 127.0.97.62:38611 requested a full tablet report, sending...
I20260929 11:25:14.884747 17559 ts_manager.cc:194] Registered new tserver with Master: 65379b598b0c4da5a60a8bc8955478bc (127.0.97.3:34549)
I20260929 11:25:14.885603 17559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49513
I20260929 11:25:14.890184 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:25:14.906131 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.894359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.0.97.62:38611)
0929 11:25:14.894529 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:14.894533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:14.894983 (+ 450us) server_negotiation.cc:407] Connection header received
0929 11:25:14.895123 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:14.895126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:14.895180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:14.895261 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:14.896649 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.897358 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.898600 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.898955 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.899039 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.901242 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:14.901246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:14.901249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:14.901277 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:14.903857 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:14.904432 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:14.904436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:14.904438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:14.904551 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:14.904936 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:14.904939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:14.904941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:14.905233 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:14.905451 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:14.905731 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 11:25:14.905875 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:25:14.910290 17559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:25:14.917069 17559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:25:14.940270 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.929275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.0.97.3:34549)
0929 11:25:14.931522 (+ 2247us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:14.931534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:14.931573 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:25:14.931688 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:14.931699 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:14.931983 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:14.932253 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:14.932992 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.934660 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.937480 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.938906 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.939045 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.939296 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:14.939751 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 11:25:14.939932 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2044,"thread_start_us":1666,"threads_started":1}
I20260929 11:25:14.940289 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.928519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34549 (local address 127.0.0.1:50968)
0929 11:25:14.929685 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:14.929745 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:14.929953 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:14.932286 (+ 2333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:14.932297 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:14.932339 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:14.932781 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.932797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.934818 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.934825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.937309 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.937323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.939116 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.939122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.939447 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.939512 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 11:25:14.939633 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 11:25:14.939815 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":51,"thread_start_us":1027,"threads_started":1}
I20260929 11:25:14.940320 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.928613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33393 (local address 127.0.0.1:51108)
0929 11:25:14.930515 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:14.930542 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:14.930655 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:14.933558 (+ 2903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:14.933565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:14.933597 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:14.934051 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.934062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.936016 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.936022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.937596 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.937609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.939495 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.939501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.939701 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.939734 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:25:14.939812 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 11:25:14.939910 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"mutex_wait_us":28,"thread_start_us":3229,"threads_started":1}
I20260929 11:25:14.941943 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.929050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.97.1:33393)
0929 11:25:14.933007 (+ 3957us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:14.933015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:14.933037 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:25:14.933144 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:14.933151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:14.933367 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:14.933554 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:14.934248 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.935864 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.937756 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.939237 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.939408 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.939613 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:14.941458 (+ 1845us) server_negotiation.cc:299] Negotiation successful
0929 11:25:14.941619 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3808,"thread_start_us":3655,"threads_started":1}
I20260929 11:25:14.942773 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.930025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44553 (local address 127.0.0.1:40556)
0929 11:25:14.934578 (+ 4553us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:14.934614 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:14.934731 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:14.935388 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:14.935393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:14.935417 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:14.935800 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.935814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.937659 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.937689 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.939769 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:14.939782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.941462 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.941469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:14.941629 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.941657 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:25:14.941747 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:25:14.942345 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4447,"mutex_wait_us":2,"thread_start_us":4338,"threads_started":1}
I20260929 11:25:14.943347 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:14.930516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.97.2:44553)
0929 11:25:14.934220 (+ 3704us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:14.934228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:14.934861 (+ 633us) server_negotiation.cc:407] Connection header received
0929 11:25:14.934964 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:14.934973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:14.935197 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:14.935380 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:14.935995 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.937514 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.939937 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:14.941244 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:14.941409 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:14.941594 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:14.941816 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 11:25:14.941980 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3538,"thread_start_us":3423,"threads_started":1}
I20260929 11:25:14.944355 17951 tablet_service.cc:1467] Processing CreateTablet for tablet 308213e7e5fc452986a4172eada90f38 (DEFAULT_TABLE table=test-table [id=50a4090db65e4111a7423af37e1dcf6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:14.945912 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308213e7e5fc452986a4172eada90f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:14.945545 17683 tablet_service.cc:1467] Processing CreateTablet for tablet 308213e7e5fc452986a4172eada90f38 (DEFAULT_TABLE table=test-table [id=50a4090db65e4111a7423af37e1dcf6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:14.946035 17817 tablet_service.cc:1467] Processing CreateTablet for tablet 308213e7e5fc452986a4172eada90f38 (DEFAULT_TABLE table=test-table [id=50a4090db65e4111a7423af37e1dcf6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:14.946903 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308213e7e5fc452986a4172eada90f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:14.947276 17817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 308213e7e5fc452986a4172eada90f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:14.958040 18034 tablet_bootstrap.cc:492] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Bootstrap starting.
I20260929 11:25:14.961969 18035 tablet_bootstrap.cc:492] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Bootstrap starting.
I20260929 11:25:14.962615 18034 tablet_bootstrap.cc:654] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:14.963657 18036 tablet_bootstrap.cc:492] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Bootstrap starting.
I20260929 11:25:14.964058 18034 log.cc:824] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:14.965394 18035 tablet_bootstrap.cc:654] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:14.966298 18035 log.cc:824] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:14.967170 18036 tablet_bootstrap.cc:654] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:14.967321 18034 tablet_bootstrap.cc:492] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: No bootstrap required, opened a new log
I20260929 11:25:14.967605 18034 ts_tablet_manager.cc:1397] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260929 11:25:14.968355 18036 log.cc:824] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:14.968901 18035 tablet_bootstrap.cc:492] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: No bootstrap required, opened a new log
I20260929 11:25:14.969179 18035 ts_tablet_manager.cc:1397] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260929 11:25:14.972285 18036 tablet_bootstrap.cc:492] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: No bootstrap required, opened a new log
I20260929 11:25:14.972520 18036 ts_tablet_manager.cc:1397] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 11:25:14.976234 18035 raft_consensus.cc:348] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:14.976593 18035 raft_consensus.cc:374] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:14.976735 18035 raft_consensus.cc:729] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65379b598b0c4da5a60a8bc8955478bc, State: Initialized, Role: FOLLOWER
I20260929 11:25:14.977324 18035 consensus_queue.cc:260] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:14.977696 18034 raft_consensus.cc:348] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:14.978106 18034 raft_consensus.cc:374] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:14.978242 18034 raft_consensus.cc:729] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325b2588397a43b69f45a6d2ae13c230, State: Initialized, Role: FOLLOWER
I20260929 11:25:14.978298 18016 heartbeater.cc:503] Master 127.0.97.62:38611 was elected leader, sending a full tablet report...
I20260929 11:25:14.978789 18034 consensus_queue.cc:260] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:14.980922 17882 heartbeater.cc:503] Master 127.0.97.62:38611 was elected leader, sending a full tablet report...
I20260929 11:25:14.981640 18034 ts_tablet_manager.cc:1428] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 11:25:14.982681 18035 ts_tablet_manager.cc:1428] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Time spent starting tablet: real 0.013s user 0.006s sys 0.003s
I20260929 11:25:14.984086 18036 raft_consensus.cc:348] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:14.984519 18036 raft_consensus.cc:374] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:14.984701 18036 raft_consensus.cc:729] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1284ed6811d04a6a8242e0e0713a47ab, State: Initialized, Role: FOLLOWER
I20260929 11:25:14.985335 18036 consensus_queue.cc:260] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:14.986805 18036 ts_tablet_manager.cc:1428] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
W20260929 11:25:15.095440 17749 tablet.cc:2369] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:15.111478 18017 tablet.cc:2369] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:15.120810 17883 tablet.cc:2369] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:15.130119 18040 raft_consensus.cc:482] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:15.130365 18040 raft_consensus.cc:504] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:15.132026 18040 leader_election.cc:288] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1284ed6811d04a6a8242e0e0713a47ab (127.0.97.1:33393), 325b2588397a43b69f45a6d2ae13c230 (127.0.97.2:44553)
I20260929 11:25:15.138494 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.132416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33393 (local address 127.0.97.3:48861)
0929 11:25:15.132858 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:15.132882 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:15.133053 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:15.133489 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:15.133492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:15.133537 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:15.133741 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:15.133755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.134922 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.134927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:15.136398 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:15.136410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.137773 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.137779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:15.137967 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.138014 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 11:25:15.138112 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 11:25:15.138252 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":1}
I20260929 11:25:15.138551 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.132384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48861 (local address 127.0.97.1:33393)
0929 11:25:15.132679 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:15.132685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:15.132988 (+ 303us) server_negotiation.cc:407] Connection header received
0929 11:25:15.133219 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:15.133225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:15.133303 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:15.133445 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:15.133879 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.134788 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.136566 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.137604 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.137704 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.137787 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:15.138149 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 11:25:15.138299 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 11:25:15.138949 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.132496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46405 (local address 127.0.97.2:44553)
0929 11:25:15.132729 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:15.132733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:15.133410 (+ 677us) server_negotiation.cc:407] Connection header received
0929 11:25:15.133607 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:15.133612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:15.133696 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:15.133791 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:15.134382 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.135255 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.137171 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.138094 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.138213 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.138293 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:15.138663 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 11:25:15.138779 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:25:15.139464 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.132448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44553 (local address 127.0.97.3:46405)
0929 11:25:15.133306 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:15.133323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:15.133462 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:15.133833 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:15.133837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:15.133861 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:15.134219 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:15.134231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.135409 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.135412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:15.137029 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:15.137044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.138295 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.138301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:15.138469 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.138503 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:25:15.139118 (+ 615us) client_negotiation.cc:240] Negotiation successful
0929 11:25:15.139238 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":461,"threads_started":1}
I20260929 11:25:15.139495 17703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308213e7e5fc452986a4172eada90f38" candidate_uuid: "65379b598b0c4da5a60a8bc8955478bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1284ed6811d04a6a8242e0e0713a47ab" is_pre_election: true
I20260929 11:25:15.140091 17703 raft_consensus.cc:2454] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65379b598b0c4da5a60a8bc8955478bc in term 0.
I20260929 11:25:15.140401 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308213e7e5fc452986a4172eada90f38" candidate_uuid: "65379b598b0c4da5a60a8bc8955478bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325b2588397a43b69f45a6d2ae13c230" is_pre_election: true
I20260929 11:25:15.140895 17905 leader_election.cc:302] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1284ed6811d04a6a8242e0e0713a47ab, 65379b598b0c4da5a60a8bc8955478bc; no voters:
I20260929 11:25:15.140902 17837 raft_consensus.cc:2454] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65379b598b0c4da5a60a8bc8955478bc in term 0.
I20260929 11:25:15.141255 18040 raft_consensus.cc:2781] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:25:15.141445 18040 raft_consensus.cc:482] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:25:15.141575 18040 raft_consensus.cc:3037] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:15.143770 18040 raft_consensus.cc:504] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:15.144357 18040 leader_election.cc:288] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [CANDIDATE]: Term 1 election: Requested vote from peers 1284ed6811d04a6a8242e0e0713a47ab (127.0.97.1:33393), 325b2588397a43b69f45a6d2ae13c230 (127.0.97.2:44553)
I20260929 11:25:15.144750 17703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308213e7e5fc452986a4172eada90f38" candidate_uuid: "65379b598b0c4da5a60a8bc8955478bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1284ed6811d04a6a8242e0e0713a47ab"
I20260929 11:25:15.144903 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308213e7e5fc452986a4172eada90f38" candidate_uuid: "65379b598b0c4da5a60a8bc8955478bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325b2588397a43b69f45a6d2ae13c230"
I20260929 11:25:15.144941 17703 raft_consensus.cc:3037] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:15.145102 17837 raft_consensus.cc:3037] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:15.148104 17837 raft_consensus.cc:2454] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65379b598b0c4da5a60a8bc8955478bc in term 1.
I20260929 11:25:15.148106 17703 raft_consensus.cc:2454] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65379b598b0c4da5a60a8bc8955478bc in term 1.
I20260929 11:25:15.148496 17907 leader_election.cc:302] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 325b2588397a43b69f45a6d2ae13c230, 65379b598b0c4da5a60a8bc8955478bc; no voters:
I20260929 11:25:15.148821 18040 raft_consensus.cc:2781] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:15.149240 18040 raft_consensus.cc:686] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 1 LEADER]: Becoming Leader. State: Replica: 65379b598b0c4da5a60a8bc8955478bc, State: Running, Role: LEADER
I20260929 11:25:15.149741 18040 consensus_queue.cc:237] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } }
I20260929 11:25:15.154124 17558 catalog_manager.cc:5495] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 65379b598b0c4da5a60a8bc8955478bc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "65379b598b0c4da5a60a8bc8955478bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65379b598b0c4da5a60a8bc8955478bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 } health_report { overall_health: UNKNOWN } } }
I20260929 11:25:15.177006 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.171944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.97.3:34549)
0929 11:25:15.172141 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:15.172145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:15.172161 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:25:15.172258 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:15.172262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:15.172333 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:15.172461 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:25:15.172843 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.173593 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.175014 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.175397 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.175491 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.175875 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:25:15.176365 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:25:15.176482 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:15.176651 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 11:25:15.176725 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:25:15.185073 17837 raft_consensus.cc:1261] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 1 FOLLOWER]: Refusing update from remote peer 65379b598b0c4da5a60a8bc8955478bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:15.185112 17703 raft_consensus.cc:1261] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 1 FOLLOWER]: Refusing update from remote peer 65379b598b0c4da5a60a8bc8955478bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:15.186053 18044 consensus_queue.cc:1035] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "325b2588397a43b69f45a6d2ae13c230" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:15.186417 18040 consensus_queue.cc:1035] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1284ed6811d04a6a8242e0e0713a47ab" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:15.212337 17559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:15.212664 17559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:15.216436 17559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d393cbb09af1449b9455caf61e61ec7e: Sending DeleteTablet for 3 replicas of tablet 308213e7e5fc452986a4172eada90f38
I20260929 11:25:15.217595 17951 tablet_service.cc:1514] Processing DeleteTablet for tablet 308213e7e5fc452986a4172eada90f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50968
I20260929 11:25:15.217698 17683 tablet_service.cc:1514] Processing DeleteTablet for tablet 308213e7e5fc452986a4172eada90f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51108
I20260929 11:25:15.217679 17817 tablet_service.cc:1514] Processing DeleteTablet for tablet 308213e7e5fc452986a4172eada90f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40556
I20260929 11:25:15.218629 18057 tablet_replica.cc:330] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: stopping tablet replica
I20260929 11:25:15.219201 18057 raft_consensus.cc:2229] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:15.219837 18057 raft_consensus.cc:2258] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:15.221161 18058 tablet_replica.cc:330] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: stopping tablet replica
I20260929 11:25:15.221246 18056 tablet_replica.cc:330] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: stopping tablet replica
I20260929 11:25:15.221618 18058 raft_consensus.cc:2229] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:15.222142 18058 raft_consensus.cc:2258] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:15.221729 18056 raft_consensus.cc:2229] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:25:15.222352 18057 ts_tablet_manager.cc:1905] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:15.222734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17620
I20260929 11:25:15.223111 18056 raft_consensus.cc:2258] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:15.225937 18056 ts_tablet_manager.cc:1905] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:15.230132 18057 ts_tablet_manager.cc:1918] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:15.231921 18058 ts_tablet_manager.cc:1905] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:15.232127 18057 log.cc:1196] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/308213e7e5fc452986a4172eada90f38
I20260929 11:25:15.232791 18057 ts_tablet_manager.cc:1939] T 308213e7e5fc452986a4172eada90f38 P 1284ed6811d04a6a8242e0e0713a47ab: Deleting consensus metadata
I20260929 11:25:15.237929 18056 ts_tablet_manager.cc:1918] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:15.238190 18056 log.cc:1196] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/308213e7e5fc452986a4172eada90f38
I20260929 11:25:15.238903 18056 ts_tablet_manager.cc:1939] T 308213e7e5fc452986a4172eada90f38 P 65379b598b0c4da5a60a8bc8955478bc: Deleting consensus metadata
I20260929 11:25:15.239078 17544 catalog_manager.cc:4841] TS 1284ed6811d04a6a8242e0e0713a47ab (127.0.97.1:33393): tablet 308213e7e5fc452986a4172eada90f38 (table test-table [id=50a4090db65e4111a7423af37e1dcf6b]) successfully deleted
I20260929 11:25:15.242182 18058 ts_tablet_manager.cc:1918] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:15.242373 18058 log.cc:1196] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/308213e7e5fc452986a4172eada90f38
I20260929 11:25:15.242913 18058 ts_tablet_manager.cc:1939] T 308213e7e5fc452986a4172eada90f38 P 325b2588397a43b69f45a6d2ae13c230: Deleting consensus metadata
I20260929 11:25:15.244558 17546 catalog_manager.cc:4841] TS 325b2588397a43b69f45a6d2ae13c230 (127.0.97.2:44553): tablet 308213e7e5fc452986a4172eada90f38 (table test-table [id=50a4090db65e4111a7423af37e1dcf6b]) successfully deleted
I20260929 11:25:15.245383 17545 catalog_manager.cc:4841] TS 65379b598b0c4da5a60a8bc8955478bc (127.0.97.3:34549): tablet 308213e7e5fc452986a4172eada90f38 (table test-table [id=50a4090db65e4111a7423af37e1dcf6b]) successfully deleted
Sep 29 11:25:15 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:25:15.256527 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.223941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.97.1:33393)
0929 11:25:15.224281 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:15.224287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:15.237160 (+ 12873us) server_negotiation.cc:407] Connection header received
0929 11:25:15.237388 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:15.237394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:15.237498 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:15.237608 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:15.239378 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.240536 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.242430 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.243089 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.243187 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.249382 (+ 6195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:15.249396 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:15.249416 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:15.249458 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:15.253353 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:15.254160 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:15.254170 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:15.254174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:15.254336 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:15.254787 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:15.254793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:15.254797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:15.255256 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:15.255556 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:15.255880 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 11:25:15.256102 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260929 11:25:15.257258 17743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:15.378073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17620
I20260929 11:25:15.402038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17754
Sep 29 11:25:15 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:25:15.418018 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.402941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.0.97.2:44553)
0929 11:25:15.403209 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:15.403215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:15.403235 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:25:15.403391 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:15.403396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:15.403486 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:15.403600 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:15.404971 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.405728 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.406971 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.407349 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.407474 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.412409 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:15.412429 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:15.412443 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:15.412472 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:15.415279 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:15.415936 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:15.415943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:15.415946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:15.416065 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:15.416452 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:15.416457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:15.416460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:15.416890 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:15.417186 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:15.417488 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 11:25:15.417695 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260929 11:25:15.418679 17877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:15.507534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17754
I20260929 11:25:15.524518 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17888
Sep 29 11:25:15 dist-test-slave-srst krb5kdc[17514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:25:15.542091 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:15.525381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.0.97.3:34549)
0929 11:25:15.525553 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:15.525560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:15.525660 (+ 100us) server_negotiation.cc:407] Connection header received
0929 11:25:15.525909 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:15.525915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:15.525981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:15.526098 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:15.527405 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.528173 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.529486 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:15.529914 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:15.530058 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:15.534635 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:15.534651 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:15.534670 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:15.534727 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:15.538945 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:15.539639 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:15.539658 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:15.539661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:15.539886 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:15.540310 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:15.540316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:15.540320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:15.540836 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:15.541151 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:15.541386 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 11:25:15.541632 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:25:15.542943 18011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:15.662848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17888
I20260929 11:25:15.682869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 17527
I20260929 11:25:15.684103 17589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:15.793547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 17527
2026-09-29T11:25:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4309 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T11:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:15Z Disabled control of system clock
I20260929 11:25:15.867585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42469
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:25:16.190320 18067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:16.190578 18067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:16.190655 18067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:16.197620 18067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:16.197726 18067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:16.197762 18067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:16.197815 18067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:16.197865 18067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:16.206535 18067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:16.207661 18067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:16.209352 18067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:16.220933 18073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:16.220991 18072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:16.221019 18075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:16.225925 18067 server_base.cc:1034] running on GCE node
I20260929 11:25:16.226658 18067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:16.227968 18067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:16.229200 18067 hybrid_clock.cc:648] HybridClock initialized: now 1790681116229174 us; error 51 us; skew 500 ppm
I20260929 11:25:16.229526 18067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:16.234982 18067 webserver.cc:460] Webserver started at http://127.0.97.62:41309/ using document root <none> and password file <none>
I20260929 11:25:16.235675 18067 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:16.235807 18067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:16.236114 18067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:16.238927 18067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "be357b2bd9a549379471ccf09f59f5a1"
format_stamp: "Formatted at 2026-09-29 11:25:16 on dist-test-slave-srst"
server_key: "7527f0ed48659ca3962d925658893ebc"
server_key_iv: "635af1fb69c7853d21d487d0433e7095"
server_key_version: "encryptionkey@0"
I20260929 11:25:16.239678 18067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "be357b2bd9a549379471ccf09f59f5a1"
format_stamp: "Formatted at 2026-09-29 11:25:16 on dist-test-slave-srst"
server_key: "7527f0ed48659ca3962d925658893ebc"
server_key_iv: "635af1fb69c7853d21d487d0433e7095"
server_key_version: "encryptionkey@0"
I20260929 11:25:16.244415 18067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260929 11:25:16.247749 18081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:16.249078 18067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 11:25:16.249264 18067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "be357b2bd9a549379471ccf09f59f5a1"
format_stamp: "Formatted at 2026-09-29 11:25:16 on dist-test-slave-srst"
server_key: "7527f0ed48659ca3962d925658893ebc"
server_key_iv: "635af1fb69c7853d21d487d0433e7095"
server_key_version: "encryptionkey@0"
I20260929 11:25:16.249475 18067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:16.273677 18067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:16.274492 18067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:16.274775 18067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:16.292922 18067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42469
I20260929 11:25:16.292939 18132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42469 every 8 connection(s)
I20260929 11:25:16.294193 18067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:16.297225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18067
I20260929 11:25:16.297390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:16.297952 388 external_mini_cluster.cc:1428] Setting key 5f0ddac7624fb689bc07b87c72a31496
I20260929 11:25:16.298120 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:16.310278 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:16.299580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.0.97.62:42469)
0929 11:25:16.300385 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:16.300395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:16.300433 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:25:16.302032 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:16.302052 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:16.302503 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:16.302898 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:16.303437 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:16.304852 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:16.306175 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:16.306833 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:16.307016 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:16.308121 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:16.308128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:16.308143 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:16.308718 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:16.308790 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:16.308814 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:16.308938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:16.309146 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 11:25:16.309407 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":250,"threads_started":1}
I20260929 11:25:16.310537 18133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1: Bootstrap starting.
I20260929 11:25:16.313560 18133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:16.314556 18133 log.cc:824] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:16.317242 18133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1: No bootstrap required, opened a new log
I20260929 11:25:16.325544 18133 raft_consensus.cc:348] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be357b2bd9a549379471ccf09f59f5a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42469 } }
I20260929 11:25:16.325924 18133 raft_consensus.cc:374] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:16.326046 18133 raft_consensus.cc:729] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be357b2bd9a549379471ccf09f59f5a1, State: Initialized, Role: FOLLOWER
I20260929 11:25:16.326615 18133 consensus_queue.cc:260] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be357b2bd9a549379471ccf09f59f5a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42469 } }
I20260929 11:25:16.326865 18133 raft_consensus.cc:388] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:16.327054 18133 raft_consensus.cc:482] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:16.327245 18133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:16.329483 18133 raft_consensus.cc:504] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be357b2bd9a549379471ccf09f59f5a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42469 } }
I20260929 11:25:16.329998 18133 leader_election.cc:302] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be357b2bd9a549379471ccf09f59f5a1; no voters:
I20260929 11:25:16.330582 18133 leader_election.cc:288] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:16.330729 18138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:16.331446 18138 raft_consensus.cc:686] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [term 1 LEADER]: Becoming Leader. State: Replica: be357b2bd9a549379471ccf09f59f5a1, State: Running, Role: LEADER
I20260929 11:25:16.331911 18138 consensus_queue.cc:237] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be357b2bd9a549379471ccf09f59f5a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42469 } }
I20260929 11:25:16.332748 18133 sys_catalog.cc:564] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:16.334774 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be357b2bd9a549379471ccf09f59f5a1. Latest consensus state: current_term: 1 leader_uuid: "be357b2bd9a549379471ccf09f59f5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be357b2bd9a549379471ccf09f59f5a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42469 } } }
I20260929 11:25:16.334800 18139 sys_catalog.cc:455] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be357b2bd9a549379471ccf09f59f5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be357b2bd9a549379471ccf09f59f5a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42469 } } }
I20260929 11:25:16.335233 18139 sys_catalog.cc:458] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:16.335230 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:16.336407 18145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:16.348428 18145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:16.356769 18145 catalog_manager.cc:1347] Generated new cluster ID: b5c54eb2006d4da5bf0ca9a73db2dcfc
I20260929 11:25:16.356895 18145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:16.370648 18145 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:16.372167 18145 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:16.385057 18145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1: Generated new TSK 0
I20260929 11:25:16.385834 18145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:16.395325 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42469
--builtin_ntp_servers=127.0.97.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:25:16.722074 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:16.722340 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:16.722494 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:16.729542 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:16.729628 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:16.729799 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:25:16.738734 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:16.740043 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:16.741890 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:16.754760 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:16.754936 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:16.754788 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:16.756016 18157 server_base.cc:1034] running on GCE node
I20260929 11:25:16.756553 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:16.757483 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:16.758687 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790681116758645 us; error 43 us; skew 500 ppm
I20260929 11:25:16.758970 18157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:16.761228 18157 webserver.cc:460] Webserver started at http://127.0.97.1:33657/ using document root <none> and password file <none>
I20260929 11:25:16.761876 18157 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:16.761962 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:16.762231 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:16.765085 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "7ef78d1da67d472d9ec9c1feb03f17b9"
format_stamp: "Formatted at 2026-09-29 11:25:16 on dist-test-slave-srst"
server_key: "9a3418c6a6dede11c7e45957c60f5364"
server_key_iv: "b01fe4507d45d7f23c484b9eba5e6477"
server_key_version: "encryptionkey@0"
I20260929 11:25:16.765836 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7ef78d1da67d472d9ec9c1feb03f17b9"
format_stamp: "Formatted at 2026-09-29 11:25:16 on dist-test-slave-srst"
server_key: "9a3418c6a6dede11c7e45957c60f5364"
server_key_iv: "b01fe4507d45d7f23c484b9eba5e6477"
server_key_version: "encryptionkey@0"
I20260929 11:25:16.770720 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:25:16.774040 18171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:16.775321 18157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:25:16.775501 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "7ef78d1da67d472d9ec9c1feb03f17b9"
format_stamp: "Formatted at 2026-09-29 11:25:16 on dist-test-slave-srst"
server_key: "9a3418c6a6dede11c7e45957c60f5364"
server_key_iv: "b01fe4507d45d7f23c484b9eba5e6477"
server_key_version: "encryptionkey@0"
I20260929 11:25:16.775735 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:16.788908 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:16.789637 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:16.789913 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:16.790835 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:16.792258 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:16.792382 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:16.792519 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:16.792692 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:16.825855 18157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43293
I20260929 11:25:16.825897 18283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43293 every 8 connection(s)
I20260929 11:25:16.827139 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:25:16.831259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18157
I20260929 11:25:16.831439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:25:16.832010 388 external_mini_cluster.cc:1428] Setting key b01e32ec8cf4f43bedce737dec25794e
I20260929 11:25:16.834846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42469
--builtin_ntp_servers=127.0.97.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:25:16.841059 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:16.829729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58103 (local address 127.0.97.62:42469)
0929 11:25:16.830216 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:16.830221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:16.831501 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 11:25:16.833665 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:16.833670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:16.833752 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:16.833892 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:16.835203 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:16.836439 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:16.838018 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:16.838700 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:16.838838 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:16.840121 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:16.840129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:16.840136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:16.840158 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:16.840222 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:16.840241 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:16.840328 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:16.840511 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 11:25:16.840645 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260929 11:25:16.841915 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:16.830339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42469 (local address 127.0.97.1:58103)
0929 11:25:16.831326 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:16.831377 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:16.833365 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:16.834285 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:16.834293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:16.834387 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:16.834987 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:16.835019 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:16.836622 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:16.836630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:16.837876 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:16.837887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:16.838902 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:16.838907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:16.839134 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:16.839830 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:16.839832 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:16.839867 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:16.839935 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:16.840332 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:16.840339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:16.840344 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:25:16.840471 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 11:25:16.840814 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
I20260929 11:25:16.843606 18284 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42469
I20260929 11:25:16.843902 18284 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:16.844738 18284 heartbeater.cc:511] Master 127.0.97.62:42469 requested a full tablet report, sending...
I20260929 11:25:16.846694 18098 ts_manager.cc:194] Registered new tserver with Master: 7ef78d1da67d472d9ec9c1feb03f17b9 (127.0.97.1:43293)
I20260929 11:25:16.848099 18098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58103
W20260929 11:25:17.153545 18288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:17.153882 18288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:17.154030 18288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:17.161144 18288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:17.161253 18288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:17.161455 18288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:25:17.170482 18288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:17.171624 18288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:17.173292 18288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:17.185910 18293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:17.185855 18294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:17.185878 18296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:17.186687 18288 server_base.cc:1034] running on GCE node
I20260929 11:25:17.187214 18288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:17.188098 18288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:17.189298 18288 hybrid_clock.cc:648] HybridClock initialized: now 1790681117189256 us; error 35 us; skew 500 ppm
I20260929 11:25:17.189636 18288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:17.191736 18288 webserver.cc:460] Webserver started at http://127.0.97.2:39797/ using document root <none> and password file <none>
I20260929 11:25:17.192394 18288 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:17.192521 18288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:17.192869 18288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:17.195683 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "2eec3c5f2e804809be97a41cd15bd55f"
format_stamp: "Formatted at 2026-09-29 11:25:17 on dist-test-slave-srst"
server_key: "d053cf1a844e5af98789a9f9f1e52a0e"
server_key_iv: "8f8b0df5114eb0be78cfde819cfd7f51"
server_key_version: "encryptionkey@0"
I20260929 11:25:17.196444 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2eec3c5f2e804809be97a41cd15bd55f"
format_stamp: "Formatted at 2026-09-29 11:25:17 on dist-test-slave-srst"
server_key: "d053cf1a844e5af98789a9f9f1e52a0e"
server_key_iv: "8f8b0df5114eb0be78cfde819cfd7f51"
server_key_version: "encryptionkey@0"
I20260929 11:25:17.201442 18288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 11:25:17.204751 18302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:17.206022 18288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:25:17.206192 18288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "2eec3c5f2e804809be97a41cd15bd55f"
format_stamp: "Formatted at 2026-09-29 11:25:17 on dist-test-slave-srst"
server_key: "d053cf1a844e5af98789a9f9f1e52a0e"
server_key_iv: "8f8b0df5114eb0be78cfde819cfd7f51"
server_key_version: "encryptionkey@0"
I20260929 11:25:17.206424 18288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:17.234059 18288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:17.234917 18288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:17.235221 18288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:17.236246 18288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:17.237731 18288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:17.237852 18288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:17.237993 18288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:17.238085 18288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:17.271032 18288 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45841
I20260929 11:25:17.271119 18414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45841 every 8 connection(s)
I20260929 11:25:17.272369 18288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:25:17.280119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18288
I20260929 11:25:17.280329 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:25:17.280912 388 external_mini_cluster.cc:1428] Setting key fa79e530ae6470d3ada383d3dbcf0024
I20260929 11:25:17.283819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42469
--builtin_ntp_servers=127.0.97.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:25:17.285773 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.275064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38805 (local address 127.0.97.62:42469)
0929 11:25:17.275255 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.275259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.276629 (+ 1370us) server_negotiation.cc:407] Connection header received
0929 11:25:17.278420 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.278426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.278513 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.278626 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:17.279807 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.280943 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.282642 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.283235 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.283362 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.284869 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:17.284878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:17.284886 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:17.284910 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:17.284955 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:17.284974 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:17.285074 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.285277 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.285386 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:25:17.286675 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.275580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42469 (local address 127.0.97.2:38805)
0929 11:25:17.276452 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.276499 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.278145 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.278877 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.278885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.278970 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:17.279605 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.279631 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.281107 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.281111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.282501 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.282519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.283440 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.283445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.283639 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.284489 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:17.284491 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:17.284529 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:17.284650 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:17.285078 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:17.285086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:17.285092 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.285234 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.285512 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260929 11:25:17.288074 18415 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42469
I20260929 11:25:17.288376 18415 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:17.289127 18415 heartbeater.cc:511] Master 127.0.97.62:42469 requested a full tablet report, sending...
I20260929 11:25:17.290514 18098 ts_manager.cc:194] Registered new tserver with Master: 2eec3c5f2e804809be97a41cd15bd55f (127.0.97.2:45841)
I20260929 11:25:17.291301 18098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38805
W20260929 11:25:17.605712 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:17.605986 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:17.606074 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:17.613312 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:17.613425 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:17.613526 18418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:25:17.622665 18418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:17.623890 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:17.625648 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:17.638337 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:17.638386 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:17.638904 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:17.639869 18418 server_base.cc:1034] running on GCE node
I20260929 11:25:17.640360 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:17.641203 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:17.642422 18418 hybrid_clock.cc:648] HybridClock initialized: now 1790681117642402 us; error 68 us; skew 500 ppm
I20260929 11:25:17.642784 18418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:17.644961 18418 webserver.cc:460] Webserver started at http://127.0.97.3:46125/ using document root <none> and password file <none>
I20260929 11:25:17.645614 18418 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:17.645735 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:17.646034 18418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:17.648846 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "58af94a8b9d24b4b9a69ac1201286dd6"
format_stamp: "Formatted at 2026-09-29 11:25:17 on dist-test-slave-srst"
server_key: "73e7065c2aeab93d67b245a8d91b5d01"
server_key_iv: "a8c2872b03241cad359a91e8335b3a7e"
server_key_version: "encryptionkey@0"
I20260929 11:25:17.649627 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "58af94a8b9d24b4b9a69ac1201286dd6"
format_stamp: "Formatted at 2026-09-29 11:25:17 on dist-test-slave-srst"
server_key: "73e7065c2aeab93d67b245a8d91b5d01"
server_key_iv: "a8c2872b03241cad359a91e8335b3a7e"
server_key_version: "encryptionkey@0"
I20260929 11:25:17.654560 18418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:25:17.657811 18432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:17.659130 18418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:25:17.659288 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "58af94a8b9d24b4b9a69ac1201286dd6"
format_stamp: "Formatted at 2026-09-29 11:25:17 on dist-test-slave-srst"
server_key: "73e7065c2aeab93d67b245a8d91b5d01"
server_key_iv: "a8c2872b03241cad359a91e8335b3a7e"
server_key_version: "encryptionkey@0"
I20260929 11:25:17.659499 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:17.681629 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:17.682396 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:17.682678 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:17.683621 18418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:17.685082 18418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:17.685204 18418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:17.685348 18418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:17.685432 18418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:17.717738 18418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43989
I20260929 11:25:17.717823 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43989 every 8 connection(s)
I20260929 11:25:17.719028 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:25:17.719597 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18418
I20260929 11:25:17.719794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:25:17.720407 388 external_mini_cluster.cc:1428] Setting key 59cd2c7600c093174d986f82f331772b
I20260929 11:25:17.731588 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.721829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58931 (local address 127.0.97.62:42469)
0929 11:25:17.722036 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.722043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.723926 (+ 1883us) server_negotiation.cc:407] Connection header received
0929 11:25:17.725705 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.725709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.725770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.725847 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:17.726783 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.727498 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.729072 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.729493 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.729608 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.730847 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:17.730852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:17.730855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:17.730871 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:17.730903 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:17.730914 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:17.730985 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.731213 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.731296 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:25:17.732487 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.722587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42469 (local address 127.0.97.3:58931)
0929 11:25:17.723786 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.723837 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.725490 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.726002 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.726009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.726096 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:17.726653 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.726668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.727627 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.727630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.728903 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.728915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.729709 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.729714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.729903 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.730567 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:17.730569 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:17.730606 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:17.730680 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:17.730998 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:17.731004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:17.731009 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.731139 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.731416 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260929 11:25:17.734138 18545 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42469
I20260929 11:25:17.734489 18545 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:17.735183 18545 heartbeater.cc:511] Master 127.0.97.62:42469 requested a full tablet report, sending...
I20260929 11:25:17.736518 18098 ts_manager.cc:194] Registered new tserver with Master: 58af94a8b9d24b4b9a69ac1201286dd6 (127.0.97.3:43989)
I20260929 11:25:17.737383 18098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:58931
I20260929 11:25:17.748203 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:25:17.760051 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.752684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.0.97.62:42469)
0929 11:25:17.752946 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.752956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.753377 (+ 421us) server_negotiation.cc:407] Connection header received
0929 11:25:17.753624 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.753648 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.753759 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.753885 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:17.754563 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.755399 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.757318 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.757821 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.757981 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.759196 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:17.759204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:17.759212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:17.759239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:17.759277 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:17.759292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:17.759402 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.759572 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.759675 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:25:17.764174 18098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:25:17.771003 18098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:25:17.794885 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.783178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.0.97.2:45841)
0929 11:25:17.783840 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.783849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.785364 (+ 1515us) server_negotiation.cc:407] Connection header received
0929 11:25:17.785769 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.785777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.786030 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.786192 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:17.787169 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.788859 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.791810 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.793352 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.793455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.793681 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.794358 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.794531 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":374,"threads_started":1}
I20260929 11:25:17.796025 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.783145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45841 (local address 127.0.0.1:50382)
0929 11:25:17.785177 (+ 2032us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.785242 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.785525 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.786551 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.786557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.786593 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:17.786965 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.786979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.789029 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.789034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.791598 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.791613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.793857 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.793863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.794072 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.794128 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.795410 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.795596 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1737,"mutex_wait_us":3,"spinlock_wait_cycles":3072,"thread_start_us":1657,"threads_started":1}
I20260929 11:25:17.796634 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.783573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42334)
0929 11:25:17.785177 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.785242 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.786353 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.787223 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.787227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.787256 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:17.787605 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.787621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.790470 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.790478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.792666 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.792681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.795778 (+ 3097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.795785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.795979 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.796025 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.796118 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.796242 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"mutex_wait_us":28,"spinlock_wait_cycles":9600,"thread_start_us":1020,"threads_started":1}
I20260929 11:25:17.797420 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.784037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.0.97.1:43293)
0929 11:25:17.784554 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.784564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.785364 (+ 800us) server_negotiation.cc:407] Connection header received
0929 11:25:17.786565 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.786574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.786797 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.786965 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:17.787802 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.789252 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.792917 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.794322 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.794439 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.794660 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.796998 (+ 2338us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.797184 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":264,"threads_started":1}
I20260929 11:25:17.801455 18219 tablet_service.cc:1467] Processing CreateTablet for tablet f0f3ee7d082148769e023307ee358562 (DEFAULT_TABLE table=test-table [id=0915dc78315e45dab2c234c1eac525db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:17.801791 18350 tablet_service.cc:1467] Processing CreateTablet for tablet f0f3ee7d082148769e023307ee358562 (DEFAULT_TABLE table=test-table [id=0915dc78315e45dab2c234c1eac525db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:17.802934 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f3ee7d082148769e023307ee358562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:17.802953 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f3ee7d082148769e023307ee358562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:17.806157 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.785545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43989 (local address 127.0.0.1:48192)
0929 11:25:17.796460 (+ 10915us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.796482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.796602 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.799682 (+ 3080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.799686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.799710 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:17.799868 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.799879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.801725 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.801730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.803407 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.803422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.805464 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.805470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.805657 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.805690 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.805777 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.805903 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10815,"thread_start_us":13074,"threads_started":1}
I20260929 11:25:17.806924 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.786127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.97.3:43989)
0929 11:25:17.798830 (+ 12703us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.798838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.798875 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:25:17.798967 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.798977 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.799199 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.799401 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:17.800039 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.801554 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.803588 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.804955 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.805099 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.805374 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.806398 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.806548 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12509,"thread_start_us":12332,"threads_started":1}
I20260929 11:25:17.810422 18480 tablet_service.cc:1467] Processing CreateTablet for tablet f0f3ee7d082148769e023307ee358562 (DEFAULT_TABLE table=test-table [id=0915dc78315e45dab2c234c1eac525db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:17.811776 18480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f3ee7d082148769e023307ee358562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:17.814921 18563 tablet_bootstrap.cc:492] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Bootstrap starting.
I20260929 11:25:17.817057 18564 tablet_bootstrap.cc:492] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Bootstrap starting.
I20260929 11:25:17.819638 18563 tablet_bootstrap.cc:654] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:17.820958 18563 log.cc:824] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:17.821202 18564 tablet_bootstrap.cc:654] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:17.822420 18564 log.cc:824] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:17.824656 18563 tablet_bootstrap.cc:492] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: No bootstrap required, opened a new log
I20260929 11:25:17.824986 18563 ts_tablet_manager.cc:1397] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260929 11:25:17.825246 18567 tablet_bootstrap.cc:492] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Bootstrap starting.
I20260929 11:25:17.825505 18564 tablet_bootstrap.cc:492] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: No bootstrap required, opened a new log
I20260929 11:25:17.825803 18564 ts_tablet_manager.cc:1397] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:25:17.829586 18567 tablet_bootstrap.cc:654] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:17.830479 18567 log.cc:824] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:17.833441 18567 tablet_bootstrap.cc:492] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: No bootstrap required, opened a new log
I20260929 11:25:17.833801 18567 ts_tablet_manager.cc:1397] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 11:25:17.836675 18563 raft_consensus.cc:348] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.837224 18563 raft_consensus.cc:374] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:17.837385 18563 raft_consensus.cc:729] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eec3c5f2e804809be97a41cd15bd55f, State: Initialized, Role: FOLLOWER
I20260929 11:25:17.838208 18563 consensus_queue.cc:260] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.838558 18564 raft_consensus.cc:348] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.839028 18564 raft_consensus.cc:374] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:17.839172 18564 raft_consensus.cc:729] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef78d1da67d472d9ec9c1feb03f17b9, State: Initialized, Role: FOLLOWER
I20260929 11:25:17.839793 18564 consensus_queue.cc:260] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.845719 18415 heartbeater.cc:503] Master 127.0.97.62:42469 was elected leader, sending a full tablet report...
I20260929 11:25:17.846720 18563 ts_tablet_manager.cc:1428] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Time spent starting tablet: real 0.022s user 0.015s sys 0.004s
I20260929 11:25:17.847442 18567 raft_consensus.cc:348] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.848086 18567 raft_consensus.cc:374] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:17.848264 18567 raft_consensus.cc:729] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58af94a8b9d24b4b9a69ac1201286dd6, State: Initialized, Role: FOLLOWER
I20260929 11:25:17.848974 18284 heartbeater.cc:503] Master 127.0.97.62:42469 was elected leader, sending a full tablet report...
I20260929 11:25:17.849152 18567 consensus_queue.cc:260] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.850999 18564 ts_tablet_manager.cc:1428] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Time spent starting tablet: real 0.025s user 0.017s sys 0.004s
I20260929 11:25:17.851507 18567 ts_tablet_manager.cc:1428] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260929 11:25:17.851572 18545 heartbeater.cc:503] Master 127.0.97.62:42469 was elected leader, sending a full tablet report...
I20260929 11:25:17.964622 18569 raft_consensus.cc:482] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:17.964933 18569 raft_consensus.cc:504] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.966560 18569 leader_election.cc:288] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58af94a8b9d24b4b9a69ac1201286dd6 (127.0.97.3:43989), 7ef78d1da67d472d9ec9c1feb03f17b9 (127.0.97.1:43293)
W20260929 11:25:17.970788 18546 tablet.cc:2369] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:17.973699 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.967101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42085 (local address 127.0.97.1:43293)
0929 11:25:17.967311 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.967319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.968061 (+ 742us) server_negotiation.cc:407] Connection header received
0929 11:25:17.968304 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.968309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.968381 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.968482 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:17.969156 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.969839 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.971923 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.972781 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.972944 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.973039 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.973316 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.973445 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:25:17.973838 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.967035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.97.2:42085)
0929 11:25:17.967944 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.967960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.968137 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.968499 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.968504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.968527 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:17.968994 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.969003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.969966 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.969974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.971729 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.971744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.972989 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.972995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.973152 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.973176 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.973287 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.973421 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":119,"thread_start_us":542,"threads_started":1}
I20260929 11:25:17.973874 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.967012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41625 (local address 127.0.97.3:43989)
0929 11:25:17.967237 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:17.967244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:17.968143 (+ 899us) server_negotiation.cc:407] Connection header received
0929 11:25:17.968320 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:17.968325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:17.968383 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:17.968485 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:17.969252 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.969963 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.972161 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.973029 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.973174 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.973270 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:17.973525 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 11:25:17.973648 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:25:17.974027 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:17.967035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43989 (local address 127.0.97.2:41625)
0929 11:25:17.968041 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:17.968060 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:17.968174 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:17.968509 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:17.968513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:17.968541 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:17.969096 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.969104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.970095 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.970104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.971992 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:17.972003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:17.973208 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:17.973214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:17.973376 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:17.973399 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:25:17.973500 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 11:25:17.973619 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":65,"spinlock_wait_cycles":27264,"thread_start_us":518,"threads_started":1}
I20260929 11:25:17.974934 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f3ee7d082148769e023307ee358562" candidate_uuid: "2eec3c5f2e804809be97a41cd15bd55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" is_pre_election: true
I20260929 11:25:17.974987 18500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f3ee7d082148769e023307ee358562" candidate_uuid: "2eec3c5f2e804809be97a41cd15bd55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" is_pre_election: true
I20260929 11:25:17.975453 18239 raft_consensus.cc:2454] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eec3c5f2e804809be97a41cd15bd55f in term 0.
I20260929 11:25:17.975461 18500 raft_consensus.cc:2454] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eec3c5f2e804809be97a41cd15bd55f in term 0.
I20260929 11:25:17.976169 18305 leader_election.cc:302] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2eec3c5f2e804809be97a41cd15bd55f, 58af94a8b9d24b4b9a69ac1201286dd6, 7ef78d1da67d472d9ec9c1feb03f17b9; no voters:
I20260929 11:25:17.976614 18569 raft_consensus.cc:2781] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:25:17.976774 18569 raft_consensus.cc:482] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:25:17.976884 18569 raft_consensus.cc:3037] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:17.979107 18569 raft_consensus.cc:504] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.979677 18569 leader_election.cc:288] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [CANDIDATE]: Term 1 election: Requested vote from peers 58af94a8b9d24b4b9a69ac1201286dd6 (127.0.97.3:43989), 7ef78d1da67d472d9ec9c1feb03f17b9 (127.0.97.1:43293)
I20260929 11:25:17.980212 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f3ee7d082148769e023307ee358562" candidate_uuid: "2eec3c5f2e804809be97a41cd15bd55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9"
I20260929 11:25:17.980211 18500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f3ee7d082148769e023307ee358562" candidate_uuid: "2eec3c5f2e804809be97a41cd15bd55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58af94a8b9d24b4b9a69ac1201286dd6"
I20260929 11:25:17.980417 18500 raft_consensus.cc:3037] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:17.980445 18239 raft_consensus.cc:3037] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:17.983630 18500 raft_consensus.cc:2454] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eec3c5f2e804809be97a41cd15bd55f in term 1.
I20260929 11:25:17.983634 18239 raft_consensus.cc:2454] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eec3c5f2e804809be97a41cd15bd55f in term 1.
I20260929 11:25:17.984040 18305 leader_election.cc:302] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2eec3c5f2e804809be97a41cd15bd55f, 58af94a8b9d24b4b9a69ac1201286dd6, 7ef78d1da67d472d9ec9c1feb03f17b9; no voters:
I20260929 11:25:17.984342 18569 raft_consensus.cc:2781] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:17.984808 18569 raft_consensus.cc:686] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 1 LEADER]: Becoming Leader. State: Replica: 2eec3c5f2e804809be97a41cd15bd55f, State: Running, Role: LEADER
I20260929 11:25:17.985313 18569 consensus_queue.cc:237] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } }
I20260929 11:25:17.989560 18098 catalog_manager.cc:5495] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eec3c5f2e804809be97a41cd15bd55f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2eec3c5f2e804809be97a41cd15bd55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eec3c5f2e804809be97a41cd15bd55f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 } health_report { overall_health: UNKNOWN } } }
W20260929 11:25:18.024837 18416 tablet.cc:2369] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:18.035789 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.029640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.0.97.2:45841)
0929 11:25:18.029884 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:18.029891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:18.030006 (+ 115us) server_negotiation.cc:407] Connection header received
0929 11:25:18.030267 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:18.030272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:18.030401 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:18.030578 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:25:18.031007 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.031749 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.033422 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.033965 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.034115 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:18.034634 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:25:18.035186 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:25:18.035352 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:18.035421 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 11:25:18.035508 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 11:25:18.044106 18239 raft_consensus.cc:1261] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 1 FOLLOWER]: Refusing update from remote peer 2eec3c5f2e804809be97a41cd15bd55f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:18.044181 18500 raft_consensus.cc:1261] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 1 FOLLOWER]: Refusing update from remote peer 2eec3c5f2e804809be97a41cd15bd55f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:18.045136 18574 consensus_queue.cc:1035] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "58af94a8b9d24b4b9a69ac1201286dd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:18.045485 18569 consensus_queue.cc:1035] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef78d1da67d472d9ec9c1feb03f17b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:18.062249 18577 mvcc.cc:205] Tried to move back new op lower bound from 7334629859499483136 to 7334629859272167424. Current Snapshot: MvccSnapshot[applied={T|T < 7334629859499483136}]
I20260929 11:25:18.074235 18098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:18.074457 18098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:18.077962 18098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be357b2bd9a549379471ccf09f59f5a1: Sending DeleteTablet for 3 replicas of tablet f0f3ee7d082148769e023307ee358562
I20260929 11:25:18.079084 18350 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f3ee7d082148769e023307ee358562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:18 UTC) from {username='slave'} at 127.0.0.1:50382
I20260929 11:25:18.079386 18219 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f3ee7d082148769e023307ee358562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:18 UTC) from {username='slave'} at 127.0.0.1:42334
I20260929 11:25:18.079386 18480 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f3ee7d082148769e023307ee358562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:18 UTC) from {username='slave'} at 127.0.0.1:48192
I20260929 11:25:18.081372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18157
I20260929 11:25:18.090590 18588 tablet_replica.cc:330] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: stopping tablet replica
I20260929 11:25:18.091231 18589 tablet_replica.cc:330] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: stopping tablet replica
I20260929 11:25:18.091982 18589 raft_consensus.cc:2229] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:18.092674 18588 raft_consensus.cc:2229] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:18.093407 18589 raft_consensus.cc:2258] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:18.094002 18588 raft_consensus.cc:2258] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:18.096517 18589 ts_tablet_manager.cc:1905] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:18.097404 18588 ts_tablet_manager.cc:1905] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:18.104426 18588 ts_tablet_manager.cc:1918] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:18.104686 18588 log.cc:1196] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/f0f3ee7d082148769e023307ee358562
I20260929 11:25:18.105283 18588 ts_tablet_manager.cc:1939] T f0f3ee7d082148769e023307ee358562 P 58af94a8b9d24b4b9a69ac1201286dd6: Deleting consensus metadata
I20260929 11:25:18.107168 18083 catalog_manager.cc:4841] TS 58af94a8b9d24b4b9a69ac1201286dd6 (127.0.97.3:43989): tablet f0f3ee7d082148769e023307ee358562 (table test-table [id=0915dc78315e45dab2c234c1eac525db]) successfully deleted
I20260929 11:25:18.109128 18587 tablet_replica.cc:330] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: stopping tablet replica
I20260929 11:25:18.109603 18587 raft_consensus.cc:2229] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:25:18.110455 18587 raft_consensus.cc:2258] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:18.111840 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.083013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.0.97.1:43293)
0929 11:25:18.083226 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:18.083232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:18.099324 (+ 16092us) server_negotiation.cc:407] Connection header received
0929 11:25:18.100943 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:18.100951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:18.101060 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:18.101167 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:18.101870 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.102952 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.107561 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.108186 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.108319 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:18.110034 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:18.110073 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:18.110091 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:18.110842 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:18.110935 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:18.110960 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:18.111118 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:18.111329 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 11:25:18.111459 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:25:18.112679 18279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:18.116379 18587 ts_tablet_manager.cc:1905] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:18.121342 18587 ts_tablet_manager.cc:1918] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:18.121515 18587 log.cc:1196] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/f0f3ee7d082148769e023307ee358562
I20260929 11:25:18.122001 18587 ts_tablet_manager.cc:1939] T f0f3ee7d082148769e023307ee358562 P 2eec3c5f2e804809be97a41cd15bd55f: Deleting consensus metadata
I20260929 11:25:18.123397 18083 catalog_manager.cc:4841] TS 2eec3c5f2e804809be97a41cd15bd55f (127.0.97.2:45841): tablet f0f3ee7d082148769e023307ee358562 (table test-table [id=0915dc78315e45dab2c234c1eac525db]) successfully deleted
I20260929 11:25:18.221204 18589 ts_tablet_manager.cc:1918] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:18.221495 18589 log.cc:1196] T f0f3ee7d082148769e023307ee358562 P 7ef78d1da67d472d9ec9c1feb03f17b9: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/f0f3ee7d082148769e023307ee358562
I20260929 11:25:18.221815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18157
W20260929 11:25:18.240078 18084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43293 (error 108)
I20260929 11:25:18.240273 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18288
I20260929 11:25:18.242270 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.241724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42354)
0929 11:25:18.241951 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.242073 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 11:25:18.242578 18084 catalog_manager.cc:4584] TS 7ef78d1da67d472d9ec9c1feb03f17b9 (127.0.97.1:43293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0f3ee7d082148769e023307ee358562: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
I20260929 11:25:18.247720 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.241277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.97.2:45841)
0929 11:25:18.241465 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:18.241469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:18.241487 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:25:18.241671 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:18.241677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:18.241809 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:18.241917 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:18.242373 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.243352 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.244731 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.245173 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.245319 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:18.246369 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:18.246379 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:18.246394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:18.246975 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:18.247027 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:18.247046 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:18.247170 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:18.247304 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 11:25:18.247398 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:25:18.248332 18410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:18.296279 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.295767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42362)
0929 11:25:18.296002 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.296120 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 11:25:18.297488 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.297024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42368)
0929 11:25:18.297183 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.297286 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:25:18.358250 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.357803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42378)
0929 11:25:18.358011 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.358102 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 11:25:18.359323 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.358995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42386)
0929 11:25:18.359119 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.359188 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:25:18.362764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18288
I20260929 11:25:18.380923 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18418
I20260929 11:25:18.387187 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.381716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.0.97.3:43989)
0929 11:25:18.381887 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:18.381892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:18.381953 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:25:18.382108 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:18.382111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:18.382199 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:18.382287 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:18.382661 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.383330 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.384665 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:18.385051 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:18.385139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:18.386036 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:18.386044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:18.386057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:18.386582 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:18.386625 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:18.386660 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:18.386746 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:18.386848 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 11:25:18.386925 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:25:18.387831 18540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:18.439715 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.439277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42388)
0929 11:25:18.439484 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.439582 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 11:25:18.440809 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:18.440352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43293 (local address 127.0.0.1:42396)
0929 11:25:18.440544 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:18.440673 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 11:25:18.474586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18418
I20260929 11:25:18.490641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18067
I20260929 11:25:18.491801 18128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:18.599568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18067
2026-09-29T11:25:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2807 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:18 dist-test-slave-srst krb5kdc[18598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:18 dist-test-slave-srst krb5kdc[18598](info): set up 2 sockets
Sep 29 11:25:18 dist-test-slave-srst krb5kdc[18598](info): commencing operation
krb5kdc: starting...
W20260929 11:25:19.700538 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:19 dist-test-slave-srst krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:25:19.935909 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39951
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:20.262199 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:20.262475 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:20.262643 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:20.270025 18611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:20.270134 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:20.270267 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:20.270370 18611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:20.270455 18611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:20.280224 18611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:20.281558 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:20.282910 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:20.294406 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:20.294420 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:20.294394 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:20.300536 18611 server_base.cc:1034] running on GCE node
I20260929 11:25:20.301381 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:20.302862 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:20.304095 18611 hybrid_clock.cc:648] HybridClock initialized: now 1790681120304048 us; error 60 us; skew 500 ppm
I20260929 11:25:20.304486 18611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:20 dist-test-slave-srst krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:20.307657 18611 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:25:20.309376 18611 webserver.cc:460] Webserver started at http://127.0.97.62:37509/ using document root <none> and password file <none>
I20260929 11:25:20.310079 18611 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:20.310225 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:20.310541 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:20.313481 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "62f33861f7924110a7b02fa567ebf263"
format_stamp: "Formatted at 2026-09-29 11:25:20 on dist-test-slave-srst"
server_key: "bf31889ec969363c81453653f98bc356"
server_key_iv: "24cef435d54b190fbb0a1037f5f86389"
server_key_version: "encryptionkey@0"
I20260929 11:25:20.314220 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "62f33861f7924110a7b02fa567ebf263"
format_stamp: "Formatted at 2026-09-29 11:25:20 on dist-test-slave-srst"
server_key: "bf31889ec969363c81453653f98bc356"
server_key_iv: "24cef435d54b190fbb0a1037f5f86389"
server_key_version: "encryptionkey@0"
I20260929 11:25:20.319303 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:20.322623 18626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:20.323899 18611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:25:20.324080 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "62f33861f7924110a7b02fa567ebf263"
format_stamp: "Formatted at 2026-09-29 11:25:20 on dist-test-slave-srst"
server_key: "bf31889ec969363c81453653f98bc356"
server_key_iv: "24cef435d54b190fbb0a1037f5f86389"
server_key_version: "encryptionkey@0"
I20260929 11:25:20.324276 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:20.373126 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:20.377883 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:20.378185 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:20.396447 18611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39951
I20260929 11:25:20.396486 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39951 every 8 connection(s)
I20260929 11:25:20.397907 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:20.401539 18678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:20.407619 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18611
I20260929 11:25:20.407791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:20.408241 388 external_mini_cluster.cc:1428] Setting key 951ba2b4e3431c16ab6f1c79d3a1e97c
I20260929 11:25:20.411590 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263: Bootstrap starting.
I20260929 11:25:20.415212 18678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:20.416182 18678 log.cc:824] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:20.419123 18678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263: No bootstrap required, opened a new log
Sep 29 11:25:20 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:20.427337 18678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f33861f7924110a7b02fa567ebf263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39951 } }
I20260929 11:25:20.427914 18678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:20.428027 18678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f33861f7924110a7b02fa567ebf263, State: Initialized, Role: FOLLOWER
I20260929 11:25:20.428733 18678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f33861f7924110a7b02fa567ebf263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39951 } }
I20260929 11:25:20.429036 18678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:20.429183 18678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:20.429376 18678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:20.430905 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:20.410027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.97.62:39951)
0929 11:25:20.410720 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:20.410731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:20.410767 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:25:20.412494 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:20.412515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:20.412994 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:20.413300 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:20.415028 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:20.416364 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:20.418090 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:20.418767 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:20.418891 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:20.423606 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:20.423613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:20.423629 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:20.423662 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:20.426763 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:20.427479 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:20.427486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:20.427491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:20.427639 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:20.428067 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:20.428071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:20.428074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:20.428770 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:20.429039 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:20.429494 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 11:25:20.429922 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":344,"threads_started":1}
I20260929 11:25:20.431753 18678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f33861f7924110a7b02fa567ebf263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39951 } }
I20260929 11:25:20.432240 18678 leader_election.cc:302] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62f33861f7924110a7b02fa567ebf263; no voters:
I20260929 11:25:20.432940 18678 leader_election.cc:288] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:20.433032 18683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:20.435369 18678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:20.435738 18683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [term 1 LEADER]: Becoming Leader. State: Replica: 62f33861f7924110a7b02fa567ebf263, State: Running, Role: LEADER
I20260929 11:25:20.436290 18683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f33861f7924110a7b02fa567ebf263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39951 } }
I20260929 11:25:20.439839 18685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62f33861f7924110a7b02fa567ebf263. Latest consensus state: current_term: 1 leader_uuid: "62f33861f7924110a7b02fa567ebf263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f33861f7924110a7b02fa567ebf263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39951 } } }
I20260929 11:25:20.439956 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62f33861f7924110a7b02fa567ebf263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f33861f7924110a7b02fa567ebf263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39951 } } }
I20260929 11:25:20.440239 18685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:20.440335 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:20.441970 18693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:20.446615 18693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:20.454909 18693 catalog_manager.cc:1347] Generated new cluster ID: ec38d446526f4b3bae63fef33d64f830
I20260929 11:25:20.455015 18693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:20.473179 18693 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:20.474996 18693 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:20.485443 18693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263: Generated new TSK 0
I20260929 11:25:20.486428 18693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:25:20.547994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39951
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:20.868026 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:20.868310 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:20.868490 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:20.875735 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:20.875909 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:20.876096 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:25:20.886111 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:20.887377 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:20.888679 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:20.901557 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:20.901522 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:20.901597 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:20.902292 18704 server_base.cc:1034] running on GCE node
I20260929 11:25:20.902872 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:20.903703 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:20.904889 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790681120904825 us; error 63 us; skew 500 ppm
I20260929 11:25:20.905190 18704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:20 dist-test-slave-srst krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:20.908123 18704 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:25:20.909472 18704 webserver.cc:460] Webserver started at http://127.0.97.1:35501/ using document root <none> and password file <none>
I20260929 11:25:20.910166 18704 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:20.910336 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:20.910626 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:20.913424 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "590c467f896c4cebb05f27f899a76174"
format_stamp: "Formatted at 2026-09-29 11:25:20 on dist-test-slave-srst"
server_key: "3088bc06a715acfd1bfe0fbdc62f3483"
server_key_iv: "98d767083b572e3481a3d9fa8156a6a9"
server_key_version: "encryptionkey@0"
I20260929 11:25:20.914194 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "590c467f896c4cebb05f27f899a76174"
format_stamp: "Formatted at 2026-09-29 11:25:20 on dist-test-slave-srst"
server_key: "3088bc06a715acfd1bfe0fbdc62f3483"
server_key_iv: "98d767083b572e3481a3d9fa8156a6a9"
server_key_version: "encryptionkey@0"
I20260929 11:25:20.918898 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:25:20.922156 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:20.923316 18704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:25:20.923488 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "590c467f896c4cebb05f27f899a76174"
format_stamp: "Formatted at 2026-09-29 11:25:20 on dist-test-slave-srst"
server_key: "3088bc06a715acfd1bfe0fbdc62f3483"
server_key_iv: "98d767083b572e3481a3d9fa8156a6a9"
server_key_version: "encryptionkey@0"
I20260929 11:25:20.923686 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:20.936609 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:20.940776 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:20.941089 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:20.941962 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:20.943215 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:20.943317 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:20.943463 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:20.943540 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:20.975716 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35291
I20260929 11:25:20.975775 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35291 every 8 connection(s)
I20260929 11:25:20.977016 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:25:20.980177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18704
I20260929 11:25:20.980355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:25:20.980994 388 external_mini_cluster.cc:1428] Setting key 1aa2962c8d3f86d731d42597ec051ea9
Sep 29 11:25:20 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:21.000790 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:20.979546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46903 (local address 127.0.97.62:39951)
0929 11:25:20.980074 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:20.980081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:20.981207 (+ 1126us) server_negotiation.cc:407] Connection header received
0929 11:25:20.983444 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:20.983448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:20.983518 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:20.983605 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:20.985755 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:20.986453 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:20.988900 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:20.989295 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:20.989390 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:20.993807 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:20.993816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:20.993824 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:20.993870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:20.997333 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:20.998287 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:20.998291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:20.998293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:20.998417 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:20.998955 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:20.998959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:20.998961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:20.999220 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:20.999328 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:21.000281 (+ 953us) server_negotiation.cc:299] Negotiation successful
0929 11:25:21.000447 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":241,"threads_started":1}
I20260929 11:25:21.002342 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:20.980119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39951 (local address 127.0.97.1:46903)
0929 11:25:20.981000 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:20.981072 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:20.983150 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:20.983893 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:20.983907 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:20.984742 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:20.985579 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:20.985607 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:20.986618 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:20.986625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:20.988761 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:20.988778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:20.989524 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:20.989533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:20.989859 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:20.990603 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:20.990638 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:20.993558 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:20.997503 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:20.997515 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:20.997535 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:20.998146 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:20.998534 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:20.998541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:20.998547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:20.998819 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:20.999346 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:20.999357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:20.999880 (+ 523us) client_negotiation.cc:769] Sending connection context
0929 11:25:21.000190 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 11:25:21.000821 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260929 11:25:21.003952 18832 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39951
I20260929 11:25:21.004258 18832 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:21.005025 18832 heartbeater.cc:511] Master 127.0.97.62:39951 requested a full tablet report, sending...
I20260929 11:25:21.007164 18643 ts_manager.cc:194] Registered new tserver with Master: 590c467f896c4cebb05f27f899a76174 (127.0.97.1:35291)
I20260929 11:25:21.008672 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46903
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:25:21.039530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39951
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:21.389149 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:21.389470 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:21.389648 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:21.396894 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:21.397040 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:21.397210 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:25:21.406508 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:21.407711 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:21.408953 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:21.422163 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:21.422420 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:21.422699 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:21.426234 18838 server_base.cc:1034] running on GCE node
I20260929 11:25:21.427042 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:21.428282 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:21.429517 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790681121429447 us; error 48 us; skew 500 ppm
I20260929 11:25:21.429931 18838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:21 dist-test-slave-srst krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:21.440557 18838 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:25:21.442276 18838 webserver.cc:460] Webserver started at http://127.0.97.2:34201/ using document root <none> and password file <none>
I20260929 11:25:21.442975 18838 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:21.443115 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:21.443426 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:21.446270 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "4b1cb82d13dc4e8fbf1353c23b7be282"
format_stamp: "Formatted at 2026-09-29 11:25:21 on dist-test-slave-srst"
server_key: "fb4dd3140e1f84342da104d6489554e6"
server_key_iv: "f4b19b42b1f051e2de20e7dc9574eefc"
server_key_version: "encryptionkey@0"
I20260929 11:25:21.447048 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4b1cb82d13dc4e8fbf1353c23b7be282"
format_stamp: "Formatted at 2026-09-29 11:25:21 on dist-test-slave-srst"
server_key: "fb4dd3140e1f84342da104d6489554e6"
server_key_iv: "f4b19b42b1f051e2de20e7dc9574eefc"
server_key_version: "encryptionkey@0"
I20260929 11:25:21.452211 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:21.455921 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:21.457468 18838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:21.457643 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "4b1cb82d13dc4e8fbf1353c23b7be282"
format_stamp: "Formatted at 2026-09-29 11:25:21 on dist-test-slave-srst"
server_key: "fb4dd3140e1f84342da104d6489554e6"
server_key_iv: "f4b19b42b1f051e2de20e7dc9574eefc"
server_key_version: "encryptionkey@0"
I20260929 11:25:21.457855 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:21.474190 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:21.478377 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:21.478698 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:21.479692 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:21.481109 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:21.481209 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:21.481365 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:21.481437 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:21.523332 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33849
I20260929 11:25:21.523352 18965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33849 every 8 connection(s)
I20260929 11:25:21.524655 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:25:21.533005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18838
I20260929 11:25:21.533206 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:25:21.533823 388 external_mini_cluster.cc:1428] Setting key d167f93e2435ae1e078b2efc62bf7ecc
Sep 29 11:25:21 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:21.549321 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:21.527615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45395 (local address 127.0.97.62:39951)
0929 11:25:21.528095 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:21.528101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:21.529337 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 11:25:21.531106 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:21.531111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:21.531214 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:21.531331 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:21.532870 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:21.533742 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:21.535727 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:21.536633 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:21.536818 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:21.541675 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:21.541685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:21.541693 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:21.541742 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:21.546250 (+ 4508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:21.546966 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:21.546971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:21.546974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:21.547116 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:21.547557 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:21.547562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:21.547567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:21.547834 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:21.547990 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:21.548707 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 11:25:21.548933 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260929 11:25:21.550376 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:21.528316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39951 (local address 127.0.97.2:45395)
0929 11:25:21.529182 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:21.529223 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:21.530845 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:21.531557 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:21.531565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:21.532060 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:21.532696 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:21.532723 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:21.533897 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:21.533902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:21.535582 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:21.535596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:21.536906 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:21.536916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:21.537251 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:21.538075 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:21.538107 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:21.541403 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:21.546404 (+ 5001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:21.546411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:21.546424 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:21.546827 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:21.547237 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:21.547242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:21.547246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:21.547433 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:21.547962 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:21.547969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:21.548322 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 11:25:21.548571 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 11:25:21.549097 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":382,"threads_started":1}
I20260929 11:25:21.551816 18966 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39951
I20260929 11:25:21.552129 18966 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:21.552934 18966 heartbeater.cc:511] Master 127.0.97.62:39951 requested a full tablet report, sending...
I20260929 11:25:21.554306 18643 ts_manager.cc:194] Registered new tserver with Master: 4b1cb82d13dc4e8fbf1353c23b7be282 (127.0.97.2:33849)
I20260929 11:25:21.555145 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45395
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:25:21.602098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39951
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:21.935111 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:21.935519 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:21.935683 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:21.943379 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:21.943536 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:21.943763 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:25:21.953701 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:21.954998 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:21.956321 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:21.968958 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:21.969025 18978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:21.969062 18980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:21.970007 18972 server_base.cc:1034] running on GCE node
I20260929 11:25:21.970608 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:21.971589 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:21.972806 18972 hybrid_clock.cc:648] HybridClock initialized: now 1790681121972775 us; error 59 us; skew 500 ppm
I20260929 11:25:21.973119 18972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:21 dist-test-slave-srst krb5kdc[18598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:21.976310 18972 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:25:21.980672 18972 webserver.cc:460] Webserver started at http://127.0.97.3:41201/ using document root <none> and password file <none>
I20260929 11:25:21.981348 18972 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:21.981469 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:21.981783 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:21.984543 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "15ee7039448a4d5698dd13e6ee0cb37f"
format_stamp: "Formatted at 2026-09-29 11:25:21 on dist-test-slave-srst"
server_key: "808b3fe5b34e6fe233c082c79744e786"
server_key_iv: "22179429d9c407387b2eae4752e025c7"
server_key_version: "encryptionkey@0"
I20260929 11:25:21.985476 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "15ee7039448a4d5698dd13e6ee0cb37f"
format_stamp: "Formatted at 2026-09-29 11:25:21 on dist-test-slave-srst"
server_key: "808b3fe5b34e6fe233c082c79744e786"
server_key_iv: "22179429d9c407387b2eae4752e025c7"
server_key_version: "encryptionkey@0"
I20260929 11:25:21.990422 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:25:21.993765 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:21.995090 18972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 11:25:21.995257 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "15ee7039448a4d5698dd13e6ee0cb37f"
format_stamp: "Formatted at 2026-09-29 11:25:21 on dist-test-slave-srst"
server_key: "808b3fe5b34e6fe233c082c79744e786"
server_key_iv: "22179429d9c407387b2eae4752e025c7"
server_key_version: "encryptionkey@0"
I20260929 11:25:21.995455 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:22.010789 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:22.011569 18832 heartbeater.cc:503] Master 127.0.97.62:39951 was elected leader, sending a full tablet report...
I20260929 11:25:22.015271 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:22.015548 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:22.016470 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:22.017866 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:22.017939 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:22.018021 18972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:22.018074 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:22.056195 18972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41685
I20260929 11:25:22.056409 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41685 every 8 connection(s)
I20260929 11:25:22.057482 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:25:22.066747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 18972
I20260929 11:25:22.066951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:25:22.067428 388 external_mini_cluster.cc:1428] Setting key aaa115cf996445c819eaa8edbd6ecdac
Sep 29 11:25:22 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:22.078038 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.060167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36529 (local address 127.0.97.62:39951)
0929 11:25:22.060774 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.060779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.061575 (+ 796us) server_negotiation.cc:407] Connection header received
0929 11:25:22.063649 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.063655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.063765 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.063897 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:22.065547 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.066453 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.068400 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.068870 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.069006 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.072951 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:22.072956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:22.072959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:22.072989 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:22.075439 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.076073 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.076077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.076079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.076187 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.076532 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.076535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.076537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.076858 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:22.076982 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.077600 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.077768 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":278,"threads_started":1}
I20260929 11:25:22.078935 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.060659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39951 (local address 127.0.97.3:36529)
0929 11:25:22.061380 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:22.061431 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:22.063366 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:22.064151 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:22.064160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:22.064772 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:22.065359 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.065376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.066704 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.066708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.068250 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.068263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.069112 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.069119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.069428 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.070288 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:22.070320 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:22.072769 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:22.075560 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:22.075567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:22.075579 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:22.075962 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:22.076273 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:22.076276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:22.076279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:22.076438 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:22.076975 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:22.076980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:22.077284 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 11:25:22.077497 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 11:25:22.077874 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":273,"threads_started":1}
I20260929 11:25:22.080209 19100 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39951
I20260929 11:25:22.080519 19100 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:22.081235 19100 heartbeater.cc:511] Master 127.0.97.62:39951 requested a full tablet report, sending...
I20260929 11:25:22.082670 18643 ts_manager.cc:194] Registered new tserver with Master: 15ee7039448a4d5698dd13e6ee0cb37f (127.0.97.3:41685)
I20260929 11:25:22.083525 18643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36529
I20260929 11:25:22.095150 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:25:22.112208 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.099444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.97.62:39951)
0929 11:25:22.099611 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.099616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.100113 (+ 497us) server_negotiation.cc:407] Connection header received
0929 11:25:22.100320 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.100325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.100408 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.100511 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:22.101914 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.102637 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.103967 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.104380 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.104517 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.107099 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:22.107104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:22.107109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:22.107147 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:22.109428 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.110165 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.110170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.110174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.110302 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.110744 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.110749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.110754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.111102 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:22.111372 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.111734 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.111928 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:25:22.116110 18643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:25:22.123052 18643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:25:22.150043 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.134978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33849 (local address 127.0.0.1:45360)
0929 11:25:22.137185 (+ 2207us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:22.137245 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:22.137502 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:22.138866 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:22.138877 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:22.138920 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:22.139510 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.139527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.142455 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.142463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.145791 (+ 3328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.145814 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.148478 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.148484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.149149 (+ 665us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.149220 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 11:25:22.149351 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 11:25:22.149576 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1863,"mutex_wait_us":393,"thread_start_us":14168,"threads_started":1}
I20260929 11:25:22.150863 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.135383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.0.97.2:33849)
0929 11:25:22.136216 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.136230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.137739 (+ 1509us) server_negotiation.cc:407] Connection header received
0929 11:25:22.137877 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.137890 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.138295 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.138525 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:22.139764 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.142229 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.146048 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.147957 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.148095 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.148322 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.150317 (+ 1995us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.150512 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":452,"threads_started":1}
I20260929 11:25:22.155066 18901 tablet_service.cc:1467] Processing CreateTablet for tablet 310f381667894cef8fe3cedbb3bbfbd8 (DEFAULT_TABLE table=test-table [id=ac7e843272c54189a176e6dfadc512c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:22.156560 18901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310f381667894cef8fe3cedbb3bbfbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:22.159401 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.136735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35291 (local address 127.0.0.1:41300)
0929 11:25:22.149302 (+ 12567us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:22.149322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:22.149439 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:22.150505 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:22.150511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:22.150542 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:22.150950 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.150965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.153209 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.153217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.155818 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.155837 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.158297 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.158304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.158569 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.158616 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 11:25:22.158714 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 11:25:22.158878 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12417,"mutex_wait_us":28,"thread_start_us":10995,"threads_started":1}
I20260929 11:25:22.160012 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.135430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41685 (local address 127.0.0.1:47436)
0929 11:25:22.148036 (+ 12606us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:22.148064 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:22.148184 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:22.149421 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:22.149425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:22.149451 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:22.149805 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.149816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.153495 (+ 3679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.153500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.156410 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.156424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.158471 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.158480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.158686 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.158721 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:25:22.158788 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 11:25:22.158911 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12463,"mutex_wait_us":1,"thread_start_us":1514,"threads_started":1}
I20260929 11:25:22.160333 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.135673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.0.97.3:41685)
0929 11:25:22.147784 (+ 12111us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.147793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.148452 (+ 659us) server_negotiation.cc:407] Connection header received
0929 11:25:22.148621 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.148631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.148998 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.150020 (+ 1022us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:22.150564 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.153283 (+ 2719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.156771 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.158178 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.159078 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.159308 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.159381 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.159856 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11898,"thread_start_us":11711,"threads_started":1}
I20260929 11:25:22.160983 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.136201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.0.97.1:35291)
0929 11:25:22.149147 (+ 12946us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.149180 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.149609 (+ 429us) server_negotiation.cc:407] Connection header received
0929 11:25:22.149723 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.149734 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.150008 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.150211 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:22.151184 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.153020 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.156040 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.157729 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.157868 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.158143 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.160431 (+ 2288us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.160651 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12342,"thread_start_us":12661,"threads_started":1}
I20260929 11:25:22.164383 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 310f381667894cef8fe3cedbb3bbfbd8 (DEFAULT_TABLE table=test-table [id=ac7e843272c54189a176e6dfadc512c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:22.164939 19035 tablet_service.cc:1467] Processing CreateTablet for tablet 310f381667894cef8fe3cedbb3bbfbd8 (DEFAULT_TABLE table=test-table [id=ac7e843272c54189a176e6dfadc512c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:22.165699 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310f381667894cef8fe3cedbb3bbfbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:22.166302 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310f381667894cef8fe3cedbb3bbfbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:22.175297 19119 tablet_bootstrap.cc:492] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Bootstrap starting.
I20260929 11:25:22.180794 19119 tablet_bootstrap.cc:654] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:22.180846 19120 tablet_bootstrap.cc:492] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Bootstrap starting.
I20260929 11:25:22.182214 19119 log.cc:824] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:22.184247 19120 tablet_bootstrap.cc:654] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:22.185858 19120 log.cc:824] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:22.185985 19119 tablet_bootstrap.cc:492] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: No bootstrap required, opened a new log
I20260929 11:25:22.186254 19119 ts_tablet_manager.cc:1397] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260929 11:25:22.189189 19122 tablet_bootstrap.cc:492] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Bootstrap starting.
I20260929 11:25:22.192694 19120 tablet_bootstrap.cc:492] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: No bootstrap required, opened a new log
I20260929 11:25:22.192958 19120 ts_tablet_manager.cc:1397] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260929 11:25:22.193024 19122 tablet_bootstrap.cc:654] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:22.194288 19122 log.cc:824] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:22.197404 19122 tablet_bootstrap.cc:492] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: No bootstrap required, opened a new log
I20260929 11:25:22.197705 19122 ts_tablet_manager.cc:1397] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 11:25:22.198199 19119 raft_consensus.cc:348] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.198653 19119 raft_consensus.cc:374] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:22.198750 19119 raft_consensus.cc:729] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1cb82d13dc4e8fbf1353c23b7be282, State: Initialized, Role: FOLLOWER
I20260929 11:25:22.199400 19119 consensus_queue.cc:260] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.205168 19120 raft_consensus.cc:348] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.205509 18966 heartbeater.cc:503] Master 127.0.97.62:39951 was elected leader, sending a full tablet report...
I20260929 11:25:22.205626 19120 raft_consensus.cc:374] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:22.205726 19120 raft_consensus.cc:729] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ee7039448a4d5698dd13e6ee0cb37f, State: Initialized, Role: FOLLOWER
I20260929 11:25:22.206351 19120 consensus_queue.cc:260] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.207852 19119 ts_tablet_manager.cc:1428] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20260929 11:25:22.208848 19100 heartbeater.cc:503] Master 127.0.97.62:39951 was elected leader, sending a full tablet report...
I20260929 11:25:22.209077 19120 ts_tablet_manager.cc:1428] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260929 11:25:22.211408 19122 raft_consensus.cc:348] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.211871 19122 raft_consensus.cc:374] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:22.211974 19122 raft_consensus.cc:729] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590c467f896c4cebb05f27f899a76174, State: Initialized, Role: FOLLOWER
I20260929 11:25:22.212563 19122 consensus_queue.cc:260] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.214248 19122 ts_tablet_manager.cc:1428] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Time spent starting tablet: real 0.016s user 0.003s sys 0.008s
W20260929 11:25:22.229298 18833 tablet.cc:2369] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:22.276762 18967 tablet.cc:2369] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:22.305104 19127 raft_consensus.cc:482] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:22.305415 19127 raft_consensus.cc:504] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.307126 19127 leader_election.cc:288] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b1cb82d13dc4e8fbf1353c23b7be282 (127.0.97.2:33849), 15ee7039448a4d5698dd13e6ee0cb37f (127.0.97.3:41685)
W20260929 11:25:22.309135 19101 tablet.cc:2369] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:22.316339 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.307650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38779 (local address 127.0.97.3:41685)
0929 11:25:22.307909 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.307918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.308758 (+ 840us) server_negotiation.cc:407] Connection header received
0929 11:25:22.309115 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.309124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.309220 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.309309 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:22.310116 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.311180 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.314092 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.315376 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.315529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.315629 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.315960 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.316094 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:25:22.316433 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.307455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41685 (local address 127.0.97.1:38779)
0929 11:25:22.308650 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:22.308678 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:22.308899 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:22.309430 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:22.309434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:22.309463 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:22.309989 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.310001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.311291 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.311296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.313940 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.313953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.315624 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.315632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.315808 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.315835 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:25:22.315915 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:25:22.316005 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":146,"thread_start_us":765,"threads_started":1}
I20260929 11:25:22.316670 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.307577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33849 (local address 127.0.97.1:39789)
0929 11:25:22.308857 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:22.308874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:22.309003 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:22.309498 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:22.309503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:22.309529 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:22.309898 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.309907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.311292 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.311296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.313939 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:22.313953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.315804 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.315810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:22.315990 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.316022 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:25:22.316111 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:25:22.316236 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":130,"thread_start_us":451,"threads_started":1}
I20260929 11:25:22.317149 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.307566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39789 (local address 127.0.97.2:33849)
0929 11:25:22.307813 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.307822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.308962 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 11:25:22.309210 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.309216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.309295 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.309371 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:22.310062 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.311159 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.314200 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.315431 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.315546 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.315629 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.316163 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.316293 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:25:22.317420 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f381667894cef8fe3cedbb3bbfbd8" candidate_uuid: "590c467f896c4cebb05f27f899a76174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" is_pre_election: true
I20260929 11:25:22.317951 19055 raft_consensus.cc:2454] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590c467f896c4cebb05f27f899a76174 in term 0.
I20260929 11:25:22.317971 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f381667894cef8fe3cedbb3bbfbd8" candidate_uuid: "590c467f896c4cebb05f27f899a76174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" is_pre_election: true
I20260929 11:25:22.318490 18921 raft_consensus.cc:2454] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590c467f896c4cebb05f27f899a76174 in term 0.
I20260929 11:25:22.318549 18721 leader_election.cc:302] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ee7039448a4d5698dd13e6ee0cb37f, 590c467f896c4cebb05f27f899a76174; no voters:
I20260929 11:25:22.318988 19127 raft_consensus.cc:2781] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:25:22.319199 19127 raft_consensus.cc:482] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:25:22.319310 19127 raft_consensus.cc:3037] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:22.321481 19127 raft_consensus.cc:504] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.322068 19127 leader_election.cc:288] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [CANDIDATE]: Term 1 election: Requested vote from peers 4b1cb82d13dc4e8fbf1353c23b7be282 (127.0.97.2:33849), 15ee7039448a4d5698dd13e6ee0cb37f (127.0.97.3:41685)
I20260929 11:25:22.322371 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f381667894cef8fe3cedbb3bbfbd8" candidate_uuid: "590c467f896c4cebb05f27f899a76174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282"
I20260929 11:25:22.322566 18921 raft_consensus.cc:3037] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:22.322540 19055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f381667894cef8fe3cedbb3bbfbd8" candidate_uuid: "590c467f896c4cebb05f27f899a76174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ee7039448a4d5698dd13e6ee0cb37f"
I20260929 11:25:22.322731 19055 raft_consensus.cc:3037] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:22.324782 18921 raft_consensus.cc:2454] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590c467f896c4cebb05f27f899a76174 in term 1.
I20260929 11:25:22.324925 19055 raft_consensus.cc:2454] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590c467f896c4cebb05f27f899a76174 in term 1.
I20260929 11:25:22.325199 18722 leader_election.cc:302] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1cb82d13dc4e8fbf1353c23b7be282, 590c467f896c4cebb05f27f899a76174; no voters:
I20260929 11:25:22.325479 19127 raft_consensus.cc:2781] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:22.325847 19127 raft_consensus.cc:686] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 1 LEADER]: Becoming Leader. State: Replica: 590c467f896c4cebb05f27f899a76174, State: Running, Role: LEADER
I20260929 11:25:22.326377 19127 consensus_queue.cc:237] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } }
I20260929 11:25:22.330722 18643 catalog_manager.cc:5495] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 reported cstate change: term changed from 0 to 1, leader changed from <none> to 590c467f896c4cebb05f27f899a76174 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "590c467f896c4cebb05f27f899a76174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590c467f896c4cebb05f27f899a76174" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35291 } health_report { overall_health: HEALTHY } } }
I20260929 11:25:22.365384 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.358892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.0.97.1:35291)
0929 11:25:22.359187 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.359195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.359213 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:25:22.359333 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.359340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.359476 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.359634 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:25:22.360167 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.361403 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.363216 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.363612 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.363740 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.364155 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:25:22.364709 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:25:22.364859 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.364967 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.365008 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 11:25:22.373797 18921 raft_consensus.cc:1261] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 1 FOLLOWER]: Refusing update from remote peer 590c467f896c4cebb05f27f899a76174: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:22.373792 19055 raft_consensus.cc:1261] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 1 FOLLOWER]: Refusing update from remote peer 590c467f896c4cebb05f27f899a76174: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:22.374850 19130 consensus_queue.cc:1035] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b1cb82d13dc4e8fbf1353c23b7be282" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:22.375231 19127 consensus_queue.cc:1035] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ee7039448a4d5698dd13e6ee0cb37f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:22.403059 18642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:22.403358 18642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:22.406802 18642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62f33861f7924110a7b02fa567ebf263: Sending DeleteTablet for 3 replicas of tablet 310f381667894cef8fe3cedbb3bbfbd8
I20260929 11:25:22.407898 18901 tablet_service.cc:1514] Processing DeleteTablet for tablet 310f381667894cef8fe3cedbb3bbfbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45360
I20260929 11:25:22.407990 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 310f381667894cef8fe3cedbb3bbfbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41300
I20260929 11:25:22.407980 19035 tablet_service.cc:1514] Processing DeleteTablet for tablet 310f381667894cef8fe3cedbb3bbfbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47436
I20260929 11:25:22.414489 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18704
I20260929 11:25:22.421458 19144 tablet_replica.cc:330] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: stopping tablet replica
I20260929 11:25:22.421981 19144 raft_consensus.cc:2229] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:25:22.422703 19144 raft_consensus.cc:2258] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:22.423549 19145 tablet_replica.cc:330] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: stopping tablet replica
I20260929 11:25:22.424175 19145 raft_consensus.cc:2229] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:22.424703 19145 raft_consensus.cc:2258] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:22.425436 19144 ts_tablet_manager.cc:1905] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:22.426743 19145 ts_tablet_manager.cc:1905] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:22.432886 19144 ts_tablet_manager.cc:1918] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:22.433139 19144 log.cc:1196] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/310f381667894cef8fe3cedbb3bbfbd8
I20260929 11:25:22.433777 19144 ts_tablet_manager.cc:1939] T 310f381667894cef8fe3cedbb3bbfbd8 P 590c467f896c4cebb05f27f899a76174: Deleting consensus metadata
I20260929 11:25:22.435741 18627 catalog_manager.cc:4841] TS 590c467f896c4cebb05f27f899a76174 (127.0.97.1:35291): tablet 310f381667894cef8fe3cedbb3bbfbd8 (table test-table [id=ac7e843272c54189a176e6dfadc512c2]) successfully deleted
I20260929 11:25:22.435292 19145 ts_tablet_manager.cc:1918] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:22.436197 19145 log.cc:1196] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/310f381667894cef8fe3cedbb3bbfbd8
I20260929 11:25:22.436790 19145 ts_tablet_manager.cc:1939] T 310f381667894cef8fe3cedbb3bbfbd8 P 15ee7039448a4d5698dd13e6ee0cb37f: Deleting consensus metadata
I20260929 11:25:22.438661 18628 catalog_manager.cc:4841] TS 15ee7039448a4d5698dd13e6ee0cb37f (127.0.97.3:41685): tablet 310f381667894cef8fe3cedbb3bbfbd8 (table test-table [id=ac7e843272c54189a176e6dfadc512c2]) successfully deleted
I20260929 11:25:22.442912 19143 tablet_replica.cc:330] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: stopping tablet replica
I20260929 11:25:22.443409 19143 raft_consensus.cc:2229] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:22.443944 19143 raft_consensus.cc:2258] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:22.445873 19143 ts_tablet_manager.cc:1905] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:25:22 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:25:22.451730 19143 ts_tablet_manager.cc:1918] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:22.451885 19143 log.cc:1196] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/310f381667894cef8fe3cedbb3bbfbd8
I20260929 11:25:22.452425 19143 ts_tablet_manager.cc:1939] T 310f381667894cef8fe3cedbb3bbfbd8 P 4b1cb82d13dc4e8fbf1353c23b7be282: Deleting consensus metadata
I20260929 11:25:22.454160 18629 catalog_manager.cc:4841] TS 4b1cb82d13dc4e8fbf1353c23b7be282 (127.0.97.2:33849): tablet 310f381667894cef8fe3cedbb3bbfbd8 (table test-table [id=ac7e843272c54189a176e6dfadc512c2]) successfully deleted
I20260929 11:25:22.456384 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.416050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.0.97.1:35291)
0929 11:25:22.416270 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.416278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.436667 (+ 20389us) server_negotiation.cc:407] Connection header received
0929 11:25:22.436872 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.436878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.436983 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.437111 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:22.438693 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.440371 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.442409 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.443230 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.443358 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.448681 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:22.448703 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:22.448724 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:22.448781 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:22.453241 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.454088 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.454096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.454099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.454288 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.454739 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.454743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.454747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.455154 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:22.455467 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.455802 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.456062 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:25:22.457093 18827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:22.558015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18704
I20260929 11:25:22.577114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18838
Sep 29 11:25:22 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:25:22.591583 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.577872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.0.97.2:33849)
0929 11:25:22.578103 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.578109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.578130 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:25:22.578207 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.578213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.578313 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.578414 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:22.579661 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.580399 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.581711 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.582098 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.582250 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.586255 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:22.586268 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:22.586289 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:22.586321 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:22.588884 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.589555 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.589562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.589564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.589682 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.590057 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.590063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.590066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.590438 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:22.590742 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.591019 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.591248 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 11:25:22.592216 18961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:22.682384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18838
I20260929 11:25:22.699733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18972
Sep 29 11:25:22 dist-test-slave-srst krb5kdc[18598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:25:22.714715 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:22.700438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.0.97.3:41685)
0929 11:25:22.700640 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:22.700647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:22.700752 (+ 105us) server_negotiation.cc:407] Connection header received
0929 11:25:22.700972 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:22.700976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:22.701054 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:22.701156 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:22.702464 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.703185 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.704469 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:22.704928 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:22.705052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:22.709243 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:22.709257 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:22.709273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:22.709306 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:22.711899 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.712525 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.712535 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.712538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.712748 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:22.713161 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:22.713166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:22.713170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:22.713526 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:22.713818 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:22.714113 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 11:25:22.714344 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:25:22.715411 19095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:22.804337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18972
I20260929 11:25:22.820817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 18611
I20260929 11:25:22.821895 18673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:22.933064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 18611
2026-09-29T11:25:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4332 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T11:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:22Z Disabled control of system clock
I20260929 11:25:23.009761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36885
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:25:23.336699 19154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:23.337001 19154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:23.337083 19154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:23.344169 19154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:23.344264 19154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:23.344304 19154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:23.344362 19154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:23.344404 19154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:23.353176 19154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:23.354476 19154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:23.356462 19154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:23.368994 19160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:23.369036 19159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:23.369077 19162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:23.369882 19154 server_base.cc:1034] running on GCE node
I20260929 11:25:23.370700 19154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:23.372087 19154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:23.373299 19154 hybrid_clock.cc:648] HybridClock initialized: now 1790681123373243 us; error 51 us; skew 500 ppm
I20260929 11:25:23.373642 19154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:23.375911 19154 webserver.cc:460] Webserver started at http://127.0.97.62:34353/ using document root <none> and password file <none>
I20260929 11:25:23.376582 19154 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:23.376744 19154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:23.377063 19154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:23.379884 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "5b649792e97442c3a1f64c49f1876f88"
format_stamp: "Formatted at 2026-09-29 11:25:23 on dist-test-slave-srst"
server_key: "948c302bee87d08a6acc75e5def41903"
server_key_iv: "c7e159ad3bcfebd87fef13f6939d1e04"
server_key_version: "encryptionkey@0"
I20260929 11:25:23.380715 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "5b649792e97442c3a1f64c49f1876f88"
format_stamp: "Formatted at 2026-09-29 11:25:23 on dist-test-slave-srst"
server_key: "948c302bee87d08a6acc75e5def41903"
server_key_iv: "c7e159ad3bcfebd87fef13f6939d1e04"
server_key_version: "encryptionkey@0"
I20260929 11:25:23.385636 19154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 11:25:23.388901 19168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:23.390125 19154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:25:23.390295 19154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "5b649792e97442c3a1f64c49f1876f88"
format_stamp: "Formatted at 2026-09-29 11:25:23 on dist-test-slave-srst"
server_key: "948c302bee87d08a6acc75e5def41903"
server_key_iv: "c7e159ad3bcfebd87fef13f6939d1e04"
server_key_version: "encryptionkey@0"
I20260929 11:25:23.390511 19154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:23.410483 19154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:23.411283 19154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:23.411568 19154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:23.430061 19154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36885
I20260929 11:25:23.430068 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36885 every 8 connection(s)
I20260929 11:25:23.431413 19154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:23.435156 19220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:23.435329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19154
I20260929 11:25:23.435490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:23.436105 388 external_mini_cluster.cc:1428] Setting key bea61a01c4adfaa040e65fcff4de3329
I20260929 11:25:23.447281 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88: Bootstrap starting.
I20260929 11:25:23.449604 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:23.438040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.0.97.62:36885)
0929 11:25:23.439069 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:23.439079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:23.439124 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:25:23.441449 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:23.441471 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:23.441990 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:23.442376 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:23.442896 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:23.444089 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:23.445421 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:23.446041 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:23.446230 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:23.447407 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:23.447416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:23.447437 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:23.447951 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:23.448021 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:23.448044 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:23.448159 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:23.448363 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 11:25:23.448699 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":389,"threads_started":1}
I20260929 11:25:23.450539 19220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:23.451541 19220 log.cc:824] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:23.454202 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88: No bootstrap required, opened a new log
I20260929 11:25:23.461854 19220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b649792e97442c3a1f64c49f1876f88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260929 11:25:23.462231 19220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:23.462344 19220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b649792e97442c3a1f64c49f1876f88, State: Initialized, Role: FOLLOWER
I20260929 11:25:23.462889 19220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b649792e97442c3a1f64c49f1876f88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260929 11:25:23.463126 19220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:23.463279 19220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:23.463449 19220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:23.465618 19220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b649792e97442c3a1f64c49f1876f88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260929 11:25:23.466091 19220 leader_election.cc:302] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b649792e97442c3a1f64c49f1876f88; no voters:
I20260929 11:25:23.466631 19220 leader_election.cc:288] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:23.466790 19225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:23.467486 19225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [term 1 LEADER]: Becoming Leader. State: Replica: 5b649792e97442c3a1f64c49f1876f88, State: Running, Role: LEADER
I20260929 11:25:23.467986 19225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b649792e97442c3a1f64c49f1876f88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260929 11:25:23.468775 19220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:23.471055 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b649792e97442c3a1f64c49f1876f88. Latest consensus state: current_term: 1 leader_uuid: "5b649792e97442c3a1f64c49f1876f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b649792e97442c3a1f64c49f1876f88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } } }
I20260929 11:25:23.470935 19226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b649792e97442c3a1f64c49f1876f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b649792e97442c3a1f64c49f1876f88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } } }
I20260929 11:25:23.471458 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:23.471504 19226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:23.472707 19233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:23.478729 19233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:23.487612 19233 catalog_manager.cc:1347] Generated new cluster ID: 1831d21209fc4ec3a17c7a1135a25fcc
I20260929 11:25:23.487717 19233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:23.506182 19233 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:23.507959 19233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:23.516682 19233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88: Generated new TSK 0
I20260929 11:25:23.517496 19233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:23.530668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--builtin_ntp_servers=127.0.97.20:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:25:23.874394 19244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:23.874667 19244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:23.874773 19244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:23.881968 19244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:23.882059 19244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:23.882225 19244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:25:23.890740 19244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:23.891959 19244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:23.893760 19244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:23.906263 19249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:23.906273 19250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:23.906930 19252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:23.907389 19244 server_base.cc:1034] running on GCE node
I20260929 11:25:23.907961 19244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:23.908839 19244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:23.910045 19244 hybrid_clock.cc:648] HybridClock initialized: now 1790681123910020 us; error 60 us; skew 500 ppm
I20260929 11:25:23.910364 19244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:23.912424 19244 webserver.cc:460] Webserver started at http://127.0.97.1:43839/ using document root <none> and password file <none>
I20260929 11:25:23.913121 19244 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:23.913247 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:23.913563 19244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:23.916353 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "ec301a4091674366b08d3b6019339697"
format_stamp: "Formatted at 2026-09-29 11:25:23 on dist-test-slave-srst"
server_key: "bac95456a8f4cd02dc55e9ae4978a022"
server_key_iv: "f70209660e1e6df6624713b60e03a41c"
server_key_version: "encryptionkey@0"
I20260929 11:25:23.917191 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ec301a4091674366b08d3b6019339697"
format_stamp: "Formatted at 2026-09-29 11:25:23 on dist-test-slave-srst"
server_key: "bac95456a8f4cd02dc55e9ae4978a022"
server_key_iv: "f70209660e1e6df6624713b60e03a41c"
server_key_version: "encryptionkey@0"
I20260929 11:25:23.922034 19244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:25:23.925315 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:23.926601 19244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:23.926784 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "ec301a4091674366b08d3b6019339697"
format_stamp: "Formatted at 2026-09-29 11:25:23 on dist-test-slave-srst"
server_key: "bac95456a8f4cd02dc55e9ae4978a022"
server_key_iv: "f70209660e1e6df6624713b60e03a41c"
server_key_version: "encryptionkey@0"
I20260929 11:25:23.926999 19244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:23.961246 19244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:23.962121 19244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:23.962401 19244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:23.963374 19244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:23.964955 19244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:23.965071 19244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:23.965193 19244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:23.965276 19244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:23.997905 19244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45743
I20260929 11:25:23.997977 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45743 every 8 connection(s)
I20260929 11:25:23.999742 19244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:25:24.009003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19244
I20260929 11:25:24.009174 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:25:24.009740 388 external_mini_cluster.cc:1428] Setting key 90e37e7c82dee728f67fc38463528a08
I20260929 11:25:24.012938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--builtin_ntp_servers=127.0.97.20:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:25:24.013489 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.002818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48131 (local address 127.0.97.62:36885)
0929 11:25:24.003660 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:24.003669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:24.004625 (+ 956us) server_negotiation.cc:407] Connection header received
0929 11:25:24.006396 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:24.006401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:24.006471 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:24.006558 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:24.007549 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.008237 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.010406 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.010940 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.011062 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.012677 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:24.012683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:24.012686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:24.012703 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:24.012756 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:24.012767 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:24.012837 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:24.013148 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 11:25:24.013240 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":576,"threads_started":1}
I20260929 11:25:24.014976 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.003493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.1:48131)
0929 11:25:24.004435 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:24.004471 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:24.006132 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:24.006760 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:24.006768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:24.006853 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:24.007416 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:24.007431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.008349 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.008353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:24.010262 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:24.010277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.011165 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.011171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:24.011451 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.012241 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:24.012246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:24.012299 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:24.012405 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:24.012872 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:24.012884 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:24.012892 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:25:24.013095 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 11:25:24.013529 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260929 11:25:24.016465 19371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260929 11:25:24.016825 19371 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:24.017547 19371 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260929 11:25:24.019598 19185 ts_manager.cc:194] Registered new tserver with Master: ec301a4091674366b08d3b6019339697 (127.0.97.1:45743)
I20260929 11:25:24.021039 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48131
W20260929 11:25:24.345963 19375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:24.346230 19375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:24.346362 19375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:24.353480 19375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:24.353578 19375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:24.353751 19375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:25:24.362733 19375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:24.363982 19375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:24.365713 19375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:24.379321 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:24.379408 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:24.379644 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:24.380393 19375 server_base.cc:1034] running on GCE node
I20260929 11:25:24.380993 19375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:24.381848 19375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:24.383037 19375 hybrid_clock.cc:648] HybridClock initialized: now 1790681124382972 us; error 49 us; skew 500 ppm
I20260929 11:25:24.383292 19375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:24.385450 19375 webserver.cc:460] Webserver started at http://127.0.97.2:34467/ using document root <none> and password file <none>
I20260929 11:25:24.386065 19375 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:24.386147 19375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:24.386427 19375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:24.389246 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "18189f3b774d465d9ee67e26d148e7ee"
format_stamp: "Formatted at 2026-09-29 11:25:24 on dist-test-slave-srst"
server_key: "59b8304d5ccde8f6577b9042d5d9e6b6"
server_key_iv: "d2fed3b740801b96e1ce498dfbcdc0d6"
server_key_version: "encryptionkey@0"
I20260929 11:25:24.389931 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "18189f3b774d465d9ee67e26d148e7ee"
format_stamp: "Formatted at 2026-09-29 11:25:24 on dist-test-slave-srst"
server_key: "59b8304d5ccde8f6577b9042d5d9e6b6"
server_key_iv: "d2fed3b740801b96e1ce498dfbcdc0d6"
server_key_version: "encryptionkey@0"
I20260929 11:25:24.395272 19375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 11:25:24.398588 19389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:24.399838 19375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:25:24.400002 19375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "18189f3b774d465d9ee67e26d148e7ee"
format_stamp: "Formatted at 2026-09-29 11:25:24 on dist-test-slave-srst"
server_key: "59b8304d5ccde8f6577b9042d5d9e6b6"
server_key_iv: "d2fed3b740801b96e1ce498dfbcdc0d6"
server_key_version: "encryptionkey@0"
I20260929 11:25:24.400218 19375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:24.419824 19375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:24.420703 19375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:24.420967 19375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:24.421892 19375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:24.423317 19375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:24.423430 19375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:24.423573 19375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:24.423655 19375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:24.456887 19375 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36967
I20260929 11:25:24.456967 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36967 every 8 connection(s)
I20260929 11:25:24.458158 19375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:25:24.467950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19375
I20260929 11:25:24.468123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:25:24.468551 388 external_mini_cluster.cc:1428] Setting key 73921a6776e7c2dc7d51ba68fff3cc9c
I20260929 11:25:24.469777 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.460701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34759 (local address 127.0.97.62:36885)
0929 11:25:24.460881 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:24.460885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:24.462172 (+ 1287us) server_negotiation.cc:407] Connection header received
0929 11:25:24.463819 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:24.463823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:24.463887 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:24.463971 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:24.465011 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.465661 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.467074 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.467426 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.467510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.468873 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:24.468881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:24.468886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:24.468905 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:24.468943 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:24.468992 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:24.469083 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:24.469394 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 11:25:24.469491 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:25:24.470906 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.461163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.2:34759)
0929 11:25:24.462044 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:24.462084 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:24.463583 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:24.464157 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:24.464164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:24.464240 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:24.464858 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:24.464875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.465805 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.465809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:24.466960 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:24.466971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.467589 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.467593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:24.467802 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.468490 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:24.468493 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:24.468562 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:24.468691 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:24.469081 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:24.469088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:24.469094 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 11:25:24.469338 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 11:25:24.469689 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":290,"threads_started":1}
I20260929 11:25:24.471675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--builtin_ntp_servers=127.0.97.20:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:25:24.472261 19502 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260929 11:25:24.472584 19502 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:24.473446 19502 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260929 11:25:24.475062 19185 ts_manager.cc:194] Registered new tserver with Master: 18189f3b774d465d9ee67e26d148e7ee (127.0.97.2:36967)
I20260929 11:25:24.476250 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34759
W20260929 11:25:24.802969 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:24.803269 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:24.803381 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:24.810621 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:24.810719 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:24.810873 19505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:25:24.819881 19505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:24.821215 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:24.822978 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:24.836458 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:24.836511 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:24.836530 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:24.837409 19505 server_base.cc:1034] running on GCE node
I20260929 11:25:24.838080 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:24.839035 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:24.840289 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790681124840226 us; error 45 us; skew 500 ppm
I20260929 11:25:24.840703 19505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:24.842942 19505 webserver.cc:460] Webserver started at http://127.0.97.3:40721/ using document root <none> and password file <none>
I20260929 11:25:24.843645 19505 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:24.843783 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:24.844131 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:24.847096 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "e0b48356796f4aff9953c7d01a41d3f8"
format_stamp: "Formatted at 2026-09-29 11:25:24 on dist-test-slave-srst"
server_key: "51a1b66c8bf963a00eb1281dccc9ef6a"
server_key_iv: "f4400e24691bfd6bbaaf4c1d84d9fd52"
server_key_version: "encryptionkey@0"
I20260929 11:25:24.847914 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e0b48356796f4aff9953c7d01a41d3f8"
format_stamp: "Formatted at 2026-09-29 11:25:24 on dist-test-slave-srst"
server_key: "51a1b66c8bf963a00eb1281dccc9ef6a"
server_key_iv: "f4400e24691bfd6bbaaf4c1d84d9fd52"
server_key_version: "encryptionkey@0"
I20260929 11:25:24.853017 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 11:25:24.856341 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:24.857635 19505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:24.857825 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "e0b48356796f4aff9953c7d01a41d3f8"
format_stamp: "Formatted at 2026-09-29 11:25:24 on dist-test-slave-srst"
server_key: "51a1b66c8bf963a00eb1281dccc9ef6a"
server_key_iv: "f4400e24691bfd6bbaaf4c1d84d9fd52"
server_key_version: "encryptionkey@0"
I20260929 11:25:24.858065 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:24.895998 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:24.897009 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:24.897361 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:24.898411 19505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:24.899968 19505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:24.900107 19505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:24.900234 19505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:24.900326 19505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:24.934924 19505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44279
I20260929 11:25:24.934973 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44279 every 8 connection(s)
I20260929 11:25:24.936226 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:25:24.936702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19505
I20260929 11:25:24.936918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:25:24.937533 388 external_mini_cluster.cc:1428] Setting key 7b8b9c46a1d3498a249b0237e6e3c540
I20260929 11:25:24.948738 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.939071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34255 (local address 127.0.97.62:36885)
0929 11:25:24.939298 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:24.939303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:24.941085 (+ 1782us) server_negotiation.cc:407] Connection header received
0929 11:25:24.942911 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:24.942916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:24.942998 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:24.943081 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:24.944055 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.944826 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.946335 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.946722 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.946811 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.948050 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:24.948054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:24.948058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:24.948074 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:24.948103 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:24.948114 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:24.948182 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:24.948374 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 11:25:24.948443 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:25:24.949692 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.939787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.3:34255)
0929 11:25:24.940936 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:24.940981 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:24.942641 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:24.943251 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:24.943259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:24.943343 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:24.943903 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:24.943919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.944951 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.944955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:24.946213 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:24.946226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.946907 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.946912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:24.947139 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.947763 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:24.947766 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:24.947802 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:24.947876 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:24.948196 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:24.948203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:24.948209 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 11:25:24.948355 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 11:25:24.948688 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":394,"threads_started":1}
I20260929 11:25:24.950953 19632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260929 11:25:24.951296 19632 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:24.951992 19632 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260929 11:25:24.953378 19185 ts_manager.cc:194] Registered new tserver with Master: e0b48356796f4aff9953c7d01a41d3f8 (127.0.97.3:44279)
I20260929 11:25:24.954089 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34255
I20260929 11:25:24.964576 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:25:24.975057 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.968656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.0.97.62:36885)
0929 11:25:24.968884 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:24.968891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:24.969264 (+ 373us) server_negotiation.cc:407] Connection header received
0929 11:25:24.969469 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:24.969476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:24.969549 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:24.969678 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:24.970178 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.970898 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.972455 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:24.972966 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:24.973110 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:24.974276 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:24.974282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:24.974285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:24.974303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:24.974334 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:24.974347 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:24.974502 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:24.974727 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 11:25:24.974785 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:25:24.978744 19185 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:25:24.985378 19185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:25:25.019132 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.998343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.97.2:36967)
0929 11:25:25.011651 (+ 13308us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.011662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.011694 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:25:25.011799 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.011807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.012019 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.012238 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:25.012989 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.014465 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.016762 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.017691 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.017856 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.018108 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.018602 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.018799 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13162,"thread_start_us":13043,"threads_started":1}
I20260929 11:25:25.019167 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.997970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36967 (local address 127.0.0.1:46440)
0929 11:25:25.010789 (+ 12819us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.010810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:25.010956 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:25.012219 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:25.012224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:25.012249 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:25.012793 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.012805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.014586 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.014593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.016524 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.016536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.017889 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.017896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.018259 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.018382 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 11:25:25.018509 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 11:25:25.018724 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12649,"mutex_wait_us":90,"thread_start_us":13088,"threads_started":1}
I20260929 11:25:25.019188 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.997335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51842)
0929 11:25:25.010107 (+ 12772us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.010141 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:25.010319 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:25.011434 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:25.011446 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:25.011488 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:25.011917 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.011930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.014264 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.014270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.015926 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.015942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.017966 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.017973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.018319 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.018387 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 11:25:25.018513 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 11:25:25.018724 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12539,"mutex_wait_us":187,"thread_start_us":12399,"threads_started":1}
I20260929 11:25:25.019989 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:24.997940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.0.97.1:45743)
0929 11:25:25.008809 (+ 10869us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.008822 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.010517 (+ 1695us) server_negotiation.cc:407] Connection header received
0929 11:25:25.010620 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.010628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.010903 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.011096 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:25.012112 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.014104 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.016191 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.017752 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.018113 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.018400 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.019452 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.019624 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10620,"thread_start_us":10467,"threads_started":1}
I20260929 11:25:25.021385 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.012036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44279 (local address 127.0.0.1:51438)
0929 11:25:25.012891 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.012909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:25.013013 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:25.013476 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:25.013481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:25.013504 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:25.014099 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.014111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.016081 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.016088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.018371 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.018387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.020570 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.020577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.020766 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.020809 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 11:25:25.020911 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:25:25.021042 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":66,"thread_start_us":542,"threads_started":1}
I20260929 11:25:25.021543 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.010867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.0.97.3:44279)
0929 11:25:25.012022 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.012030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.013105 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 11:25:25.013178 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.013182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.013320 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.013488 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:25.014319 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.015939 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.018634 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.020039 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.020186 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.020454 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.020973 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.021164 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":852,"threads_started":1}
I20260929 11:25:25.023345 19437 tablet_service.cc:1467] Processing CreateTablet for tablet 778c0ff66017463eaa46aa94b4bd1835 (DEFAULT_TABLE table=test-table [id=8c2aecdfc078485f9af6b92148302d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:25.023343 19306 tablet_service.cc:1467] Processing CreateTablet for tablet 778c0ff66017463eaa46aa94b4bd1835 (DEFAULT_TABLE table=test-table [id=8c2aecdfc078485f9af6b92148302d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:25.024415 19437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778c0ff66017463eaa46aa94b4bd1835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:25.024415 19371 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260929 11:25:25.024529 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778c0ff66017463eaa46aa94b4bd1835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:25.028069 19567 tablet_service.cc:1467] Processing CreateTablet for tablet 778c0ff66017463eaa46aa94b4bd1835 (DEFAULT_TABLE table=test-table [id=8c2aecdfc078485f9af6b92148302d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:25.029510 19567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778c0ff66017463eaa46aa94b4bd1835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:25.037529 19650 tablet_bootstrap.cc:492] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Bootstrap starting.
I20260929 11:25:25.040311 19651 tablet_bootstrap.cc:492] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: Bootstrap starting.
I20260929 11:25:25.040856 19650 tablet_bootstrap.cc:654] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:25.042119 19650 log.cc:824] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:25.043054 19652 tablet_bootstrap.cc:492] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Bootstrap starting.
I20260929 11:25:25.044065 19651 tablet_bootstrap.cc:654] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:25.045255 19651 log.cc:824] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:25.046176 19652 tablet_bootstrap.cc:654] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:25.046391 19650 tablet_bootstrap.cc:492] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: No bootstrap required, opened a new log
I20260929 11:25:25.046663 19650 ts_tablet_manager.cc:1397] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 11:25:25.047348 19652 log.cc:824] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:25.048395 19651 tablet_bootstrap.cc:492] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: No bootstrap required, opened a new log
I20260929 11:25:25.048710 19651 ts_tablet_manager.cc:1397] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:25:25.050249 19652 tablet_bootstrap.cc:492] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: No bootstrap required, opened a new log
I20260929 11:25:25.050521 19652 ts_tablet_manager.cc:1397] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 11:25:25.054299 19650 raft_consensus.cc:348] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.054673 19650 raft_consensus.cc:374] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:25.054791 19650 raft_consensus.cc:729] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18189f3b774d465d9ee67e26d148e7ee, State: Initialized, Role: FOLLOWER
I20260929 11:25:25.055383 19650 consensus_queue.cc:260] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.056453 19502 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260929 11:25:25.057153 19650 ts_tablet_manager.cc:1428] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260929 11:25:25.059134 19651 raft_consensus.cc:348] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.059563 19651 raft_consensus.cc:374] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:25.059664 19651 raft_consensus.cc:729] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec301a4091674366b08d3b6019339697, State: Initialized, Role: FOLLOWER
I20260929 11:25:25.060237 19651 consensus_queue.cc:260] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.060981 19652 raft_consensus.cc:348] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.061380 19652 raft_consensus.cc:374] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:25.061481 19652 raft_consensus.cc:729] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b48356796f4aff9953c7d01a41d3f8, State: Initialized, Role: FOLLOWER
I20260929 11:25:25.062151 19651 ts_tablet_manager.cc:1428] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 11:25:25.062034 19652 consensus_queue.cc:260] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.063310 19632 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260929 11:25:25.063668 19652 ts_tablet_manager.cc:1428] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260929 11:25:25.188354 19633 tablet.cc:2369] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:25.210228 19503 tablet.cc:2369] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:25.252519 19372 tablet.cc:2369] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:25.304266 19656 raft_consensus.cc:482] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:25.304688 19656 raft_consensus.cc:504] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.306365 19656 leader_election.cc:288] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec301a4091674366b08d3b6019339697 (127.0.97.1:45743), e0b48356796f4aff9953c7d01a41d3f8 (127.0.97.3:44279)
I20260929 11:25:25.313135 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.306708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53487 (local address 127.0.97.1:45743)
0929 11:25:25.306958 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.306968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.307452 (+ 484us) server_negotiation.cc:407] Connection header received
0929 11:25:25.307709 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.307716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.307806 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.307873 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:25.308659 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.309444 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.311280 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.312111 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.312209 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.312279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.312736 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.312886 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:25:25.313153 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.306664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.97.2:53487)
0929 11:25:25.307322 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.307337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:25.307519 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:25.308069 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:25.308074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:25.308103 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:25.308508 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.308521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.309573 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.309579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.311086 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.311098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.312303 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.312313 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.312543 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.312566 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:25:25.312688 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 11:25:25.312801 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":2,"thread_start_us":377,"threads_started":1}
I20260929 11:25:25.313180 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.307640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42835 (local address 127.0.97.3:44279)
0929 11:25:25.307821 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.307826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.308058 (+ 232us) server_negotiation.cc:407] Connection header received
0929 11:25:25.308173 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.308181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.308258 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.308352 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:25.308967 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.309722 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.311460 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.312241 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.312363 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.312458 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.312834 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.312972 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:25:25.313253 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.307402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44279 (local address 127.0.97.2:42835)
0929 11:25:25.307954 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.307967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:25.308069 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:25.308371 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:25.308374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:25.308394 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:25.308815 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.308827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.309849 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.309853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.311275 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:25.311289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.312420 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.312424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:25.312647 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.312700 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 11:25:25.312792 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 11:25:25.312913 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":53,"thread_start_us":301,"threads_started":1}
I20260929 11:25:25.314211 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c0ff66017463eaa46aa94b4bd1835" candidate_uuid: "18189f3b774d465d9ee67e26d148e7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b48356796f4aff9953c7d01a41d3f8" is_pre_election: true
I20260929 11:25:25.314210 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c0ff66017463eaa46aa94b4bd1835" candidate_uuid: "18189f3b774d465d9ee67e26d148e7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec301a4091674366b08d3b6019339697" is_pre_election: true
I20260929 11:25:25.314668 19326 raft_consensus.cc:2454] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18189f3b774d465d9ee67e26d148e7ee in term 0.
I20260929 11:25:25.314671 19587 raft_consensus.cc:2454] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18189f3b774d465d9ee67e26d148e7ee in term 0.
I20260929 11:25:25.315270 19393 leader_election.cc:302] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18189f3b774d465d9ee67e26d148e7ee, ec301a4091674366b08d3b6019339697; no voters:
I20260929 11:25:25.315641 19656 raft_consensus.cc:2781] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:25:25.315785 19656 raft_consensus.cc:482] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:25:25.315896 19656 raft_consensus.cc:3037] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:25.318127 19656 raft_consensus.cc:504] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.318708 19656 leader_election.cc:288] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [CANDIDATE]: Term 1 election: Requested vote from peers ec301a4091674366b08d3b6019339697 (127.0.97.1:45743), e0b48356796f4aff9953c7d01a41d3f8 (127.0.97.3:44279)
I20260929 11:25:25.319114 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c0ff66017463eaa46aa94b4bd1835" candidate_uuid: "18189f3b774d465d9ee67e26d148e7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec301a4091674366b08d3b6019339697"
I20260929 11:25:25.319169 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c0ff66017463eaa46aa94b4bd1835" candidate_uuid: "18189f3b774d465d9ee67e26d148e7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b48356796f4aff9953c7d01a41d3f8"
I20260929 11:25:25.319330 19326 raft_consensus.cc:3037] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:25.319347 19587 raft_consensus.cc:3037] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:25.322495 19587 raft_consensus.cc:2454] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18189f3b774d465d9ee67e26d148e7ee in term 1.
I20260929 11:25:25.322490 19326 raft_consensus.cc:2454] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18189f3b774d465d9ee67e26d148e7ee in term 1.
I20260929 11:25:25.322850 19393 leader_election.cc:302] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18189f3b774d465d9ee67e26d148e7ee, ec301a4091674366b08d3b6019339697; no voters:
I20260929 11:25:25.323108 19656 raft_consensus.cc:2781] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:25.323544 19656 raft_consensus.cc:686] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 1 LEADER]: Becoming Leader. State: Replica: 18189f3b774d465d9ee67e26d148e7ee, State: Running, Role: LEADER
I20260929 11:25:25.324008 19656 consensus_queue.cc:237] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } }
I20260929 11:25:25.328843 19184 catalog_manager.cc:5495] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 18189f3b774d465d9ee67e26d148e7ee (127.0.97.2). New cstate: current_term: 1 leader_uuid: "18189f3b774d465d9ee67e26d148e7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18189f3b774d465d9ee67e26d148e7ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36967 } health_report { overall_health: HEALTHY } } }
I20260929 11:25:25.374634 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.368012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.0.97.2:36967)
0929 11:25:25.368183 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.368190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.368277 (+ 87us) server_negotiation.cc:407] Connection header received
0929 11:25:25.368505 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.368509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.368665 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.368839 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:25:25.369291 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.370364 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.372466 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.373077 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.373172 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.373577 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:25:25.374125 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:25:25.374257 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.374393 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.374432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:25:25.383008 19587 raft_consensus.cc:1261] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 1 FOLLOWER]: Refusing update from remote peer 18189f3b774d465d9ee67e26d148e7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:25.383008 19326 raft_consensus.cc:1261] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697 [term 1 FOLLOWER]: Refusing update from remote peer 18189f3b774d465d9ee67e26d148e7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:25.383975 19661 consensus_queue.cc:1035] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b48356796f4aff9953c7d01a41d3f8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:25.384308 19656 consensus_queue.cc:1035] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec301a4091674366b08d3b6019339697" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:25.396806 19665 mvcc.cc:205] Tried to move back new op lower bound from 7334629889558700032 to 7334629889331777536. Current Snapshot: MvccSnapshot[applied={T|T < 7334629889558700032}]
I20260929 11:25:25.399879 19666 mvcc.cc:205] Tried to move back new op lower bound from 7334629889558700032 to 7334629889331777536. Current Snapshot: MvccSnapshot[applied={T|T < 7334629889558700032}]
I20260929 11:25:25.411373 19184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:25.411648 19184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:25.415263 19184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b649792e97442c3a1f64c49f1876f88: Sending DeleteTablet for 3 replicas of tablet 778c0ff66017463eaa46aa94b4bd1835
I20260929 11:25:25.416426 19437 tablet_service.cc:1514] Processing DeleteTablet for tablet 778c0ff66017463eaa46aa94b4bd1835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:25 UTC) from {username='slave'} at 127.0.0.1:46440
I20260929 11:25:25.416486 19306 tablet_service.cc:1514] Processing DeleteTablet for tablet 778c0ff66017463eaa46aa94b4bd1835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:25 UTC) from {username='slave'} at 127.0.0.1:51842
I20260929 11:25:25.416757 19567 tablet_service.cc:1514] Processing DeleteTablet for tablet 778c0ff66017463eaa46aa94b4bd1835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:25 UTC) from {username='slave'} at 127.0.0.1:51438
I20260929 11:25:25.417642 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19244
I20260929 11:25:25.425700 19675 tablet_replica.cc:330] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: stopping tablet replica
I20260929 11:25:25.426260 19675 raft_consensus.cc:2229] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:25.426868 19675 raft_consensus.cc:2258] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:25.427770 19673 tablet_replica.cc:330] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: stopping tablet replica
I20260929 11:25:25.428223 19673 raft_consensus.cc:2229] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:25:25.429010 19673 raft_consensus.cc:2258] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:25.429399 19675 ts_tablet_manager.cc:1905] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:25.434456 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.419228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.0.97.1:45743)
0929 11:25:25.419449 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.419458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.425257 (+ 5799us) server_negotiation.cc:407] Connection header received
0929 11:25:25.425676 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.425682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.425813 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.425892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:25.426670 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.427823 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.429987 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.430680 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.430775 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.432706 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:25.432716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:25.432733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:25.433525 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:25.433587 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:25.433614 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:25.433712 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.433954 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.434066 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:25:25.435710 19366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:25.442416 19673 ts_tablet_manager.cc:1905] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:25.446301 19675 ts_tablet_manager.cc:1918] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:25.452219 19673 ts_tablet_manager.cc:1918] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:25.452421 19673 log.cc:1196] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/778c0ff66017463eaa46aa94b4bd1835
I20260929 11:25:25.452663 19675 log.cc:1196] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/778c0ff66017463eaa46aa94b4bd1835
I20260929 11:25:25.453465 19675 ts_tablet_manager.cc:1939] T 778c0ff66017463eaa46aa94b4bd1835 P e0b48356796f4aff9953c7d01a41d3f8: Deleting consensus metadata
I20260929 11:25:25.453632 19673 ts_tablet_manager.cc:1939] T 778c0ff66017463eaa46aa94b4bd1835 P 18189f3b774d465d9ee67e26d148e7ee: Deleting consensus metadata
I20260929 11:25:25.455449 19170 catalog_manager.cc:4841] TS 18189f3b774d465d9ee67e26d148e7ee (127.0.97.2:36967): tablet 778c0ff66017463eaa46aa94b4bd1835 (table test-table [id=8c2aecdfc078485f9af6b92148302d83]) successfully deleted
I20260929 11:25:25.455518 19172 catalog_manager.cc:4841] TS e0b48356796f4aff9953c7d01a41d3f8 (127.0.97.3:44279): tablet 778c0ff66017463eaa46aa94b4bd1835 (table test-table [id=8c2aecdfc078485f9af6b92148302d83]) successfully deleted
I20260929 11:25:25.550756 19674 tablet_replica.cc:330] T 778c0ff66017463eaa46aa94b4bd1835 P ec301a4091674366b08d3b6019339697: stopping tablet replica
I20260929 11:25:25.550806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19244
I20260929 11:25:25.568367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19375
W20260929 11:25:25.568378 19172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45743 (error 108)
I20260929 11:25:25.571385 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.570618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51852)
0929 11:25:25.571090 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.571211 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
W20260929 11:25:25.571682 19172 catalog_manager.cc:4584] TS ec301a4091674366b08d3b6019339697 (127.0.97.1:45743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 778c0ff66017463eaa46aa94b4bd1835: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
I20260929 11:25:25.577272 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.569243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.97.2:36967)
0929 11:25:25.569423 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.569430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.569894 (+ 464us) server_negotiation.cc:407] Connection header received
0929 11:25:25.569984 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.569990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.570093 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.570222 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:25.570741 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.571931 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.573586 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.574257 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.574385 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.575559 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:25.575571 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:25.575590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:25.576415 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:25.576476 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:25.576500 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:25.576626 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.576810 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.576913 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:25:25.578111 19497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:25.625557 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.624975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51858)
0929 11:25:25.625237 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.625359 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 11:25:25.626724 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.626269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51868)
0929 11:25:25.626465 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.626559 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 11:25:25.687211 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.686616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51880)
0929 11:25:25.686874 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.686993 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 11:25:25.688779 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.688149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51886)
0929 11:25:25.688514 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.688624 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250}
I20260929 11:25:25.689530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19375
I20260929 11:25:25.707405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19505
I20260929 11:25:25.714211 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.708091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.0.97.3:44279)
0929 11:25:25.708269 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:25.708276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:25.708376 (+ 100us) server_negotiation.cc:407] Connection header received
0929 11:25:25.708577 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:25.708581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:25.708706 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:25.708826 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:25.709197 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.709883 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.711200 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:25.711583 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:25.711710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:25.712889 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:25.712899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:25.712914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:25.713467 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:25.713518 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:25.713537 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:25.713651 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:25.713782 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 11:25:25.713901 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:25:25.714818 19627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:25.769409 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.768925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51894)
0929 11:25:25.769117 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.769206 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:25:25.770457 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:25.770060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45743 (local address 127.0.0.1:51898)
0929 11:25:25.770261 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:25.770332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 11:25:25.802930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19505
I20260929 11:25:25.819890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19154
I20260929 11:25:25.821008 19215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:25.927834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19154
2026-09-29T11:25:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2993 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:26 dist-test-slave-srst krb5kdc[19685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:26 dist-test-slave-srst krb5kdc[19685](info): set up 2 sockets
Sep 29 11:25:26 dist-test-slave-srst krb5kdc[19685](info): commencing operation
krb5kdc: starting...
W20260929 11:25:27.016662 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:27 dist-test-slave-srst krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:25:27.245599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43415
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:27.563768 19698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:27.564052 19698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:27.564138 19698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:27.571242 19698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:27.571360 19698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:27.571403 19698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:27.571453 19698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:27.571487 19698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:27.582158 19698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:27.583375 19698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:27.584623 19698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:27.596033 19704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:27.596112 19703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:27.596086 19706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:27.597209 19698 server_base.cc:1034] running on GCE node
I20260929 11:25:27.597882 19698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:27.599157 19698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:27.600339 19698 hybrid_clock.cc:648] HybridClock initialized: now 1790681127600290 us; error 48 us; skew 500 ppm
I20260929 11:25:27.600708 19698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:27 dist-test-slave-srst krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:27.607605 19698 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:25:27.609207 19698 webserver.cc:460] Webserver started at http://127.0.97.62:43509/ using document root <none> and password file <none>
I20260929 11:25:27.609922 19698 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:27.610062 19698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:27.610369 19698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:27.613301 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "6dc7dcb7163f4564b78894e6efdb6e03"
format_stamp: "Formatted at 2026-09-29 11:25:27 on dist-test-slave-srst"
server_key: "46bbfd1fcddd5fe4ab399f99bfb85cfb"
server_key_iv: "fe0b4e03a03dad46aed71885666771be"
server_key_version: "encryptionkey@0"
I20260929 11:25:27.614037 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "6dc7dcb7163f4564b78894e6efdb6e03"
format_stamp: "Formatted at 2026-09-29 11:25:27 on dist-test-slave-srst"
server_key: "46bbfd1fcddd5fe4ab399f99bfb85cfb"
server_key_iv: "fe0b4e03a03dad46aed71885666771be"
server_key_version: "encryptionkey@0"
I20260929 11:25:27.618973 19698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:25:27.622227 19713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:27.623427 19698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:25:27.623592 19698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "6dc7dcb7163f4564b78894e6efdb6e03"
format_stamp: "Formatted at 2026-09-29 11:25:27 on dist-test-slave-srst"
server_key: "46bbfd1fcddd5fe4ab399f99bfb85cfb"
server_key_iv: "fe0b4e03a03dad46aed71885666771be"
server_key_version: "encryptionkey@0"
I20260929 11:25:27.623773 19698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:27.647410 19698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:27.651319 19698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:27.651579 19698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:27.669526 19698 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43415
I20260929 11:25:27.669577 19764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43415 every 8 connection(s)
I20260929 11:25:27.670879 19698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:27.674386 19765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:27.676674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19698
I20260929 11:25:27.676842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:27.677373 388 external_mini_cluster.cc:1428] Setting key 6c91d735e7f775ce8113b5b3959276d1
I20260929 11:25:27.686643 19765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03: Bootstrap starting.
I20260929 11:25:27.690409 19765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03: Neither blocks nor log segments found. Creating new log.
Sep 29 11:25:27 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:27.691576 19765 log.cc:824] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:27.694345 19765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03: No bootstrap required, opened a new log
I20260929 11:25:27.699625 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:27.679166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.0.97.62:43415)
0929 11:25:27.679947 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:27.679958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:27.679997 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:25:27.681661 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:27.681683 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:27.682092 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:27.682430 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:27.684135 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:27.685419 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:27.687057 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:27.687667 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:27.687791 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:27.692713 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:27.692724 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:27.692738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:27.692771 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:27.695664 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:27.696329 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:27.696341 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:27.696346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:27.696543 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:27.696985 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:27.696990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:27.696997 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:27.697475 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:27.697791 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:27.698236 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 11:25:27.698661 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":374,"threads_started":1}
I20260929 11:25:27.702023 19765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43415 } }
I20260929 11:25:27.702417 19765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:27.702519 19765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc7dcb7163f4564b78894e6efdb6e03, State: Initialized, Role: FOLLOWER
I20260929 11:25:27.702993 19765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43415 } }
I20260929 11:25:27.703191 19765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:27.703286 19765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:27.703403 19765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:27.705469 19765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43415 } }
I20260929 11:25:27.705874 19765 leader_election.cc:302] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dc7dcb7163f4564b78894e6efdb6e03; no voters:
I20260929 11:25:27.706354 19765 leader_election.cc:288] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:27.706487 19770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:27.707185 19770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc7dcb7163f4564b78894e6efdb6e03, State: Running, Role: LEADER
I20260929 11:25:27.707728 19770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43415 } }
I20260929 11:25:27.708564 19765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:27.710487 19772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc7dcb7163f4564b78894e6efdb6e03. Latest consensus state: current_term: 1 leader_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43415 } } }
I20260929 11:25:27.710918 19772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:27.713126 19778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:27.713080 19771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc7dcb7163f4564b78894e6efdb6e03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43415 } } }
I20260929 11:25:27.713456 19771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:27.718309 19778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:27.726729 19778 catalog_manager.cc:1347] Generated new cluster ID: 21b767691e4c4fa68b24b558627d9499
I20260929 11:25:27.726855 19778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:27.746152 19778 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:27.747967 19778 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:27.754814 19778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03: Generated new TSK 0
I20260929 11:25:27.755808 19778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 11:25:27.820029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43415
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:28.152019 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:28.152331 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:28.152493 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:28.159449 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:28.159579 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:28.159741 19791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:25:28.168926 19791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:28.170176 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:28.171486 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:28.184062 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:28.184092 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:28.184295 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:28.184917 19791 server_base.cc:1034] running on GCE node
I20260929 11:25:28.185487 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:28.186345 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:28.187564 19791 hybrid_clock.cc:648] HybridClock initialized: now 1790681128187522 us; error 58 us; skew 500 ppm
I20260929 11:25:28.187858 19791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:28 dist-test-slave-srst krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:28.190901 19791 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 11:25:28.192354 19791 webserver.cc:460] Webserver started at http://127.0.97.1:33971/ using document root <none> and password file <none>
I20260929 11:25:28.193091 19791 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:28.193228 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:28.193548 19791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:28.196372 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "aff54d08d12249488faa2f1fa72c3016"
format_stamp: "Formatted at 2026-09-29 11:25:28 on dist-test-slave-srst"
server_key: "39dc8c1b8d4c624a653893f93a8455a3"
server_key_iv: "92d34f5823164289956a29098360da12"
server_key_version: "encryptionkey@0"
I20260929 11:25:28.197191 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "aff54d08d12249488faa2f1fa72c3016"
format_stamp: "Formatted at 2026-09-29 11:25:28 on dist-test-slave-srst"
server_key: "39dc8c1b8d4c624a653893f93a8455a3"
server_key_iv: "92d34f5823164289956a29098360da12"
server_key_version: "encryptionkey@0"
I20260929 11:25:28.202172 19791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:25:28.205654 19806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:28.206957 19791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:25:28.207129 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "aff54d08d12249488faa2f1fa72c3016"
format_stamp: "Formatted at 2026-09-29 11:25:28 on dist-test-slave-srst"
server_key: "39dc8c1b8d4c624a653893f93a8455a3"
server_key_iv: "92d34f5823164289956a29098360da12"
server_key_version: "encryptionkey@0"
I20260929 11:25:28.207327 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:28.225652 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:28.229650 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:28.229890 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:28.230742 19791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:28.231986 19791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:28.232059 19791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:28.232142 19791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:28.232185 19791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:28.262502 19791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33339
I20260929 11:25:28.262549 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33339 every 8 connection(s)
I20260929 11:25:28.263736 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:25:28.265385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19791
I20260929 11:25:28.265553 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:25:28.266134 388 external_mini_cluster.cc:1428] Setting key 13f6a631a76648604f12b9d310ae7f89
Sep 29 11:25:28 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:28.286342 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:28.266412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36499 (local address 127.0.97.62:43415)
0929 11:25:28.267216 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:28.267222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:28.268017 (+ 795us) server_negotiation.cc:407] Connection header received
0929 11:25:28.270833 (+ 2816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:28.270837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:28.270904 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:28.270995 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:28.273070 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.273807 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.276184 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.276583 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.276747 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:28.280523 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:28.280528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:28.280532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:28.280564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:28.283135 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:28.284018 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:28.284022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:28.284024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:28.284137 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:28.284630 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:28.284635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:28.284638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:28.284885 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:28.285015 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:28.285904 (+ 889us) server_negotiation.cc:299] Negotiation successful
0929 11:25:28.286074 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":603,"threads_started":1}
I20260929 11:25:28.287909 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:28.267007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43415 (local address 127.0.97.1:36499)
0929 11:25:28.267852 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:28.267900 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:28.270538 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:28.271272 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:28.271285 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:28.272048 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:28.272888 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:28.272915 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.273955 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.273962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:28.276055 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:28.276071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.276870 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.276880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:28.277208 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:28.277948 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:28.277973 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:28.280314 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:28.283291 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:28.283302 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:28.283321 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:28.283912 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:28.284246 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:28.284253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:28.284260 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:28.284494 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:28.285024 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:28.285035 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:28.285513 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 11:25:28.285833 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 11:25:28.286419 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":312,"threads_started":1}
I20260929 11:25:28.289474 19919 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43415
I20260929 11:25:28.289772 19919 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:28.290570 19919 heartbeater.cc:511] Master 127.0.97.62:43415 requested a full tablet report, sending...
I20260929 11:25:28.292742 19730 ts_manager.cc:194] Registered new tserver with Master: aff54d08d12249488faa2f1fa72c3016 (127.0.97.1:33339)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 11:25:28.294252 19730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 11:25:28.323173 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43415
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:28.642235 19925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:28.642580 19925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:28.642748 19925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:28.649893 19925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:28.650053 19925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:28.650295 19925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:25:28.660086 19925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:28.661425 19925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:28.662737 19925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:28.677465 19931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:28.677903 19930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:28.678095 19933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:28.678601 19925 server_base.cc:1034] running on GCE node
I20260929 11:25:28.679147 19925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:28.680063 19925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:28.681290 19925 hybrid_clock.cc:648] HybridClock initialized: now 1790681128681238 us; error 40 us; skew 500 ppm
I20260929 11:25:28.681632 19925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:28 dist-test-slave-srst krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:28.684760 19925 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 11:25:28.686255 19925 webserver.cc:460] Webserver started at http://127.0.97.2:33369/ using document root <none> and password file <none>
I20260929 11:25:28.686957 19925 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:28.687079 19925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:28.687377 19925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:28.690248 19925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "5dfc0ca752e8486fbbf92911291aef0c"
format_stamp: "Formatted at 2026-09-29 11:25:28 on dist-test-slave-srst"
server_key: "69baa77ba128a4cba4c8e724e4457ce9"
server_key_iv: "0095745af60871c4a2b017253ec7537c"
server_key_version: "encryptionkey@0"
I20260929 11:25:28.691020 19925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5dfc0ca752e8486fbbf92911291aef0c"
format_stamp: "Formatted at 2026-09-29 11:25:28 on dist-test-slave-srst"
server_key: "69baa77ba128a4cba4c8e724e4457ce9"
server_key_iv: "0095745af60871c4a2b017253ec7537c"
server_key_version: "encryptionkey@0"
I20260929 11:25:28.696071 19925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:28.699404 19940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:28.700732 19925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:25:28.700891 19925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "5dfc0ca752e8486fbbf92911291aef0c"
format_stamp: "Formatted at 2026-09-29 11:25:28 on dist-test-slave-srst"
server_key: "69baa77ba128a4cba4c8e724e4457ce9"
server_key_iv: "0095745af60871c4a2b017253ec7537c"
server_key_version: "encryptionkey@0"
I20260929 11:25:28.701079 19925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:28.712970 19925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:28.717208 19925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:28.717494 19925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:28.718413 19925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:28.719779 19925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:28.719872 19925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:28.720014 19925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:28.720086 19925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:28.752691 19925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42833
I20260929 11:25:28.752775 20052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42833 every 8 connection(s)
I20260929 11:25:28.754063 19925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:25:28.756450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 19925
I20260929 11:25:28.756654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:25:28.757176 388 external_mini_cluster.cc:1428] Setting key 43908d518b028ee18ee2cd0ece6f56c3
Sep 29 11:25:28 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:28.777961 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:28.756617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54847 (local address 127.0.97.62:43415)
0929 11:25:28.756830 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:28.756838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:28.758168 (+ 1330us) server_negotiation.cc:407] Connection header received
0929 11:25:28.760381 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:28.760386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:28.760454 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:28.760537 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:28.762683 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.763350 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.765830 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.766240 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.766331 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:28.770565 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:28.770574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:28.770582 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:28.770626 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:28.774425 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:28.775226 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:28.775234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:28.775241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:28.775402 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:28.775821 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:28.775828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:28.775833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:28.776243 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:28.776413 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:28.777208 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 11:25:28.777491 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:25:28.778638 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:28.757155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43415 (local address 127.0.97.2:54847)
0929 11:25:28.757971 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:28.758031 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:28.760092 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:28.760846 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:28.760875 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:28.761670 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:28.762513 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:28.762542 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.763506 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.763514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:28.765675 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:28.765694 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:28.766466 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:28.766475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:28.766845 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:28.767657 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:28.767691 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:28.770323 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:28.774579 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:28.774586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:28.774599 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:28.775086 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:28.775511 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:28.775514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:28.775516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:28.775706 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:28.776399 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:28.776405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:28.776810 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 11:25:28.777053 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 11:25:28.777487 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":315,"threads_started":1}
I20260929 11:25:28.780314 20053 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43415
I20260929 11:25:28.780678 20053 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:28.781396 20053 heartbeater.cc:511] Master 127.0.97.62:43415 requested a full tablet report, sending...
I20260929 11:25:28.783052 19730 ts_manager.cc:194] Registered new tserver with Master: 5dfc0ca752e8486fbbf92911291aef0c (127.0.97.2:42833)
I20260929 11:25:28.783900 19730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54847
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 11:25:28.816256 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43415
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:29.140738 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:29.141031 20058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:29.141122 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:29.148054 20058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:29.148152 20058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:29.148258 20058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:25:29.157882 20058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:29.159192 20058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:29.160470 20058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:29.173072 20063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:29.173009 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:29.173110 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:29.173974 20058 server_base.cc:1034] running on GCE node
I20260929 11:25:29.174613 20058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:29.175515 20058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:29.176679 20058 hybrid_clock.cc:648] HybridClock initialized: now 1790681129176623 us; error 47 us; skew 500 ppm
I20260929 11:25:29.176997 20058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:29 dist-test-slave-srst krb5kdc[19685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:29.180126 20058 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 11:25:29.181535 20058 webserver.cc:460] Webserver started at http://127.0.97.3:46209/ using document root <none> and password file <none>
I20260929 11:25:29.182149 20058 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:29.182242 20058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:29.182488 20058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:29.185369 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "e92c9cdac40e42c08223b1a63b62222a"
format_stamp: "Formatted at 2026-09-29 11:25:29 on dist-test-slave-srst"
server_key: "8a0b88ed473825d18867dd90feda80e3"
server_key_iv: "8c18331576ca6147eda52eff03606a75"
server_key_version: "encryptionkey@0"
I20260929 11:25:29.186092 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e92c9cdac40e42c08223b1a63b62222a"
format_stamp: "Formatted at 2026-09-29 11:25:29 on dist-test-slave-srst"
server_key: "8a0b88ed473825d18867dd90feda80e3"
server_key_iv: "8c18331576ca6147eda52eff03606a75"
server_key_version: "encryptionkey@0"
I20260929 11:25:29.191395 20058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:29.194671 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:29.196015 20058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:25:29.196172 20058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "e92c9cdac40e42c08223b1a63b62222a"
format_stamp: "Formatted at 2026-09-29 11:25:29 on dist-test-slave-srst"
server_key: "8a0b88ed473825d18867dd90feda80e3"
server_key_iv: "8c18331576ca6147eda52eff03606a75"
server_key_version: "encryptionkey@0"
I20260929 11:25:29.196378 20058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:29.214249 20058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:29.218665 20058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:29.218955 20058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:29.219905 20058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:29.221359 20058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:29.221433 20058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:29.221572 20058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:29.221623 20058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:29.254312 20058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45529
I20260929 11:25:29.254405 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45529 every 8 connection(s)
I20260929 11:25:29.255590 20058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:25:29.259831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20058
I20260929 11:25:29.260002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:25:29.260548 388 external_mini_cluster.cc:1428] Setting key a021a2c76d120ffba24df7bad4f0aac9
Sep 29 11:25:29 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:29.276467 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.258111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54279 (local address 127.0.97.62:43415)
0929 11:25:29.258281 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.258285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.259283 (+ 998us) server_negotiation.cc:407] Connection header received
0929 11:25:29.261771 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.261776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.261856 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.261959 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:29.264101 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.264924 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.267133 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.267614 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.267706 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.270971 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:29.270976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:29.270980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:29.271022 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:29.273536 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.274233 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.274241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.274246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.274372 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.274780 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.274787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.274790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.275067 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:29.275246 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.275899 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.276121 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:25:29.277431 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.258450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43415 (local address 127.0.97.3:54279)
0929 11:25:29.259121 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:29.259158 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:29.261463 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:29.262417 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:29.262429 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:29.263209 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:29.263931 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.263953 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.265067 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.265073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.266998 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.267013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.267847 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.267855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.268081 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.268734 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:25:29.268760 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:29.270791 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:29.273679 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:29.273685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:29.273697 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:29.274075 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:29.274481 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:25:29.274486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:25:29.274490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:25:29.274670 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:25:29.275232 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:29.275239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:29.275536 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 11:25:29.275802 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 11:25:29.276310 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260929 11:25:29.278699 20186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43415
I20260929 11:25:29.278999 20186 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:29.279700 20186 heartbeater.cc:511] Master 127.0.97.62:43415 requested a full tablet report, sending...
I20260929 11:25:29.281111 19730 ts_manager.cc:194] Registered new tserver with Master: e92c9cdac40e42c08223b1a63b62222a (127.0.97.3:45529)
I20260929 11:25:29.281908 19730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54279
I20260929 11:25:29.287446 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:25:29.297255 19919 heartbeater.cc:503] Master 127.0.97.62:43415 was elected leader, sending a full tablet report...
I20260929 11:25:29.304190 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.291310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.0.97.62:43415)
0929 11:25:29.291471 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.291476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.291932 (+ 456us) server_negotiation.cc:407] Connection header received
0929 11:25:29.292127 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.292133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.292203 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.292304 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:29.293733 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.294427 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.295685 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.296140 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.296273 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.298969 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:29.298975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:29.298980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:29.299016 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:29.301552 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.302197 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.302203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.302207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.302331 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.302684 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.302689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.302692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.303053 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:29.303316 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.303642 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.303877 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:25:29.307801 19730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:25:29.315026 19730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:25:29.347677 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.327032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.97.3:45529)
0929 11:25:29.339254 (+ 12222us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.339261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.339288 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:25:29.339363 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.339368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.339568 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.339757 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:29.340421 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.342114 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.345002 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.346391 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.346509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.346736 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.347212 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.347370 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12074,"thread_start_us":12945,"threads_started":1}
I20260929 11:25:29.347903 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.326649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45529 (local address 127.0.0.1:45878)
0929 11:25:29.337549 (+ 10900us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:29.337585 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:29.337765 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:29.339757 (+ 1992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:29.339764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:29.339807 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:29.340188 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.340206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.342277 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.342282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.344829 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.344844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.346689 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.346695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.346991 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.347028 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:25:29.347164 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:25:29.347339 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10740,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":10367,"threads_started":1}
I20260929 11:25:29.351131 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.326766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.97.2:42833)
0929 11:25:29.340611 (+ 13845us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.340619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.340645 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:25:29.340749 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.340758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.340982 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.342131 (+ 1149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:29.343134 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.344750 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.348257 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.349677 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.349825 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.350018 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.350657 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.350839 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13642,"thread_start_us":13513,"threads_started":1}
I20260929 11:25:29.351751 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.326618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42833 (local address 127.0.0.1:60330)
0929 11:25:29.337543 (+ 10925us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:29.337588 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:29.337775 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:29.342386 (+ 4611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:29.342392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:29.342419 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:29.342920 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.342932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.344901 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.344909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.348097 (+ 3188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.348110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.349899 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.349908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.350330 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.350367 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:25:29.350465 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 11:25:29.351359 (+ 894us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10572,"mutex_wait_us":851,"spinlock_wait_cycles":19840,"thread_start_us":19731,"threads_started":1}
I20260929 11:25:29.351879 20121 tablet_service.cc:1467] Processing CreateTablet for tablet fa8ed8cdab374587b21e20b7fab14b26 (DEFAULT_TABLE table=test-table [id=59217351166746d5b6e4813cc265d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:29.353253 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8ed8cdab374587b21e20b7fab14b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:29.354087 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.328826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33339 (local address 127.0.0.1:36390)
0929 11:25:29.346682 (+ 17856us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:29.346699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:29.346814 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:29.347577 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:29.347582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:29.347605 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:29.347968 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.347979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.349917 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.349922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.351940 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.351954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.353343 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.353349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.353509 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.353557 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 11:25:29.353659 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:25:29.353789 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17678,"thread_start_us":8873,"threads_started":1}
I20260929 11:25:29.354182 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.329612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.97.1:33339)
0929 11:25:29.345940 (+ 16328us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.345951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.346837 (+ 886us) server_negotiation.cc:407] Connection header received
0929 11:25:29.346944 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.346951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.347162 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.347325 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:29.348165 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.349756 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.352130 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.353157 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.353299 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.353511 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.353701 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.353868 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16100,"thread_start_us":16330,"threads_started":1}
I20260929 11:25:29.357784 19988 tablet_service.cc:1467] Processing CreateTablet for tablet fa8ed8cdab374587b21e20b7fab14b26 (DEFAULT_TABLE table=test-table [id=59217351166746d5b6e4813cc265d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:29.357784 19854 tablet_service.cc:1467] Processing CreateTablet for tablet fa8ed8cdab374587b21e20b7fab14b26 (DEFAULT_TABLE table=test-table [id=59217351166746d5b6e4813cc265d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:29.359138 19854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8ed8cdab374587b21e20b7fab14b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:29.359141 19988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8ed8cdab374587b21e20b7fab14b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:29.364678 20204 tablet_bootstrap.cc:492] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Bootstrap starting.
I20260929 11:25:29.368806 20204 tablet_bootstrap.cc:654] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:29.369810 20204 log.cc:824] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:29.372280 20204 tablet_bootstrap.cc:492] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: No bootstrap required, opened a new log
I20260929 11:25:29.372548 20204 ts_tablet_manager.cc:1397] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 11:25:29.373497 20207 tablet_bootstrap.cc:492] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Bootstrap starting.
I20260929 11:25:29.374611 20206 tablet_bootstrap.cc:492] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Bootstrap starting.
I20260929 11:25:29.378235 20207 tablet_bootstrap.cc:654] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:29.379125 20206 tablet_bootstrap.cc:654] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:29.379472 20207 log.cc:824] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:29.380170 20206 log.cc:824] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:29.382848 20207 tablet_bootstrap.cc:492] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: No bootstrap required, opened a new log
I20260929 11:25:29.383108 20207 ts_tablet_manager.cc:1397] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 11:25:29.383155 20206 tablet_bootstrap.cc:492] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: No bootstrap required, opened a new log
I20260929 11:25:29.383378 20206 ts_tablet_manager.cc:1397] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 11:25:29.383565 20204 raft_consensus.cc:348] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.383932 20204 raft_consensus.cc:374] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:29.384047 20204 raft_consensus.cc:729] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c9cdac40e42c08223b1a63b62222a, State: Initialized, Role: FOLLOWER
I20260929 11:25:29.384701 20204 consensus_queue.cc:260] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.389281 20186 heartbeater.cc:503] Master 127.0.97.62:43415 was elected leader, sending a full tablet report...
I20260929 11:25:29.390125 20204 ts_tablet_manager.cc:1428] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 11:25:29.393903 20206 raft_consensus.cc:348] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.394339 20206 raft_consensus.cc:374] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:29.394436 20206 raft_consensus.cc:729] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff54d08d12249488faa2f1fa72c3016, State: Initialized, Role: FOLLOWER
I20260929 11:25:29.394472 20207 raft_consensus.cc:348] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.394994 20207 raft_consensus.cc:374] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:29.395099 20207 raft_consensus.cc:729] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfc0ca752e8486fbbf92911291aef0c, State: Initialized, Role: FOLLOWER
I20260929 11:25:29.395128 20206 consensus_queue.cc:260] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.395735 20207 consensus_queue.cc:260] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.396944 20053 heartbeater.cc:503] Master 127.0.97.62:43415 was elected leader, sending a full tablet report...
I20260929 11:25:29.397533 20207 ts_tablet_manager.cc:1428] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 11:25:29.398372 20206 ts_tablet_manager.cc:1428] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260929 11:25:29.400969 20212 raft_consensus.cc:482] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:29.401201 20212 raft_consensus.cc:504] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.402801 20212 leader_election.cc:288] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92c9cdac40e42c08223b1a63b62222a (127.0.97.3:45529), aff54d08d12249488faa2f1fa72c3016 (127.0.97.1:33339)
I20260929 11:25:29.409340 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.403755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50265 (local address 127.0.97.1:33339)
0929 11:25:29.403918 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.403923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.404202 (+ 279us) server_negotiation.cc:407] Connection header received
0929 11:25:29.404375 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.404380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.404441 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.404533 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:29.405188 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.405875 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.407636 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.408394 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.408535 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.408671 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.408961 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.409080 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 11:25:29.409416 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.403182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33339 (local address 127.0.97.2:50265)
0929 11:25:29.404109 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:29.404123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:29.404258 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:29.404568 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:29.404573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:29.404632 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:29.405048 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.405058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.406005 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.406010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.407483 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.407494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.408632 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.408638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.408807 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.408830 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:25:29.408927 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:25:29.409044 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":441,"threads_started":1}
I20260929 11:25:29.409483 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.403127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50367 (local address 127.0.97.3:45529)
0929 11:25:29.403378 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.403384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.403907 (+ 523us) server_negotiation.cc:407] Connection header received
0929 11:25:29.404148 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.404154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.404224 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.404338 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:29.404941 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.405775 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.407672 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.408507 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.408678 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.408782 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.409048 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.409217 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:25:29.409582 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.402951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45529 (local address 127.0.97.2:50367)
0929 11:25:29.403775 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:29.403804 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:29.404010 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:29.404361 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:29.404366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:29.404392 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:29.404785 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.404795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.405896 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.405904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.407524 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:29.407536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.408752 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.408762 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:29.408899 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.408921 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:25:29.409035 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 11:25:29.409154 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":365,"threads_started":1}
I20260929 11:25:29.410612 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ed8cdab374587b21e20b7fab14b26" candidate_uuid: "5dfc0ca752e8486fbbf92911291aef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c9cdac40e42c08223b1a63b62222a" is_pre_election: true
I20260929 11:25:29.411119 20141 raft_consensus.cc:2454] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dfc0ca752e8486fbbf92911291aef0c in term 0.
I20260929 11:25:29.411006 19874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ed8cdab374587b21e20b7fab14b26" candidate_uuid: "5dfc0ca752e8486fbbf92911291aef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff54d08d12249488faa2f1fa72c3016" is_pre_election: true
I20260929 11:25:29.411525 19874 raft_consensus.cc:2454] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dfc0ca752e8486fbbf92911291aef0c in term 0.
I20260929 11:25:29.411777 19941 leader_election.cc:302] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfc0ca752e8486fbbf92911291aef0c, e92c9cdac40e42c08223b1a63b62222a; no voters:
I20260929 11:25:29.412199 20212 raft_consensus.cc:2781] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:25:29.412364 20212 raft_consensus.cc:482] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:25:29.412470 20212 raft_consensus.cc:3037] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:29.415091 20212 raft_consensus.cc:504] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.415663 20212 leader_election.cc:288] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [CANDIDATE]: Term 1 election: Requested vote from peers e92c9cdac40e42c08223b1a63b62222a (127.0.97.3:45529), aff54d08d12249488faa2f1fa72c3016 (127.0.97.1:33339)
I20260929 11:25:29.416123 19874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ed8cdab374587b21e20b7fab14b26" candidate_uuid: "5dfc0ca752e8486fbbf92911291aef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff54d08d12249488faa2f1fa72c3016"
I20260929 11:25:29.416123 20141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8ed8cdab374587b21e20b7fab14b26" candidate_uuid: "5dfc0ca752e8486fbbf92911291aef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c9cdac40e42c08223b1a63b62222a"
I20260929 11:25:29.416323 20141 raft_consensus.cc:3037] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:29.416323 19874 raft_consensus.cc:3037] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:29.419523 20141 raft_consensus.cc:2454] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dfc0ca752e8486fbbf92911291aef0c in term 1.
I20260929 11:25:29.419523 19874 raft_consensus.cc:2454] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dfc0ca752e8486fbbf92911291aef0c in term 1.
I20260929 11:25:29.419898 19941 leader_election.cc:302] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfc0ca752e8486fbbf92911291aef0c, e92c9cdac40e42c08223b1a63b62222a; no voters:
I20260929 11:25:29.420171 20212 raft_consensus.cc:2781] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:29.420640 20212 raft_consensus.cc:686] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 1 LEADER]: Becoming Leader. State: Replica: 5dfc0ca752e8486fbbf92911291aef0c, State: Running, Role: LEADER
I20260929 11:25:29.421097 20212 consensus_queue.cc:237] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } }
I20260929 11:25:29.425508 19730 catalog_manager.cc:5495] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dfc0ca752e8486fbbf92911291aef0c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5dfc0ca752e8486fbbf92911291aef0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dfc0ca752e8486fbbf92911291aef0c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260929 11:25:29.444823 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.439736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.0.97.2:42833)
0929 11:25:29.439943 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.439951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.440019 (+ 68us) server_negotiation.cc:407] Connection header received
0929 11:25:29.440203 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.440207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.440297 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.440414 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:25:29.440795 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.441503 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.442835 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.443218 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.443346 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.443770 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:25:29.444262 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:25:29.444370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.444438 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.444498 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:25:29.452565 19874 raft_consensus.cc:1261] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 1 FOLLOWER]: Refusing update from remote peer 5dfc0ca752e8486fbbf92911291aef0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:29.452668 20141 raft_consensus.cc:1261] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 1 FOLLOWER]: Refusing update from remote peer 5dfc0ca752e8486fbbf92911291aef0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:29.453569 20215 consensus_queue.cc:1035] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff54d08d12249488faa2f1fa72c3016" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:29.453879 20212 consensus_queue.cc:1035] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92c9cdac40e42c08223b1a63b62222a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:29.478820 19728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:29.479139 19728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:29.482859 19728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dc7dcb7163f4564b78894e6efdb6e03: Sending DeleteTablet for 3 replicas of tablet fa8ed8cdab374587b21e20b7fab14b26
I20260929 11:25:29.483964 20121 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8ed8cdab374587b21e20b7fab14b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45878
I20260929 11:25:29.483997 19988 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8ed8cdab374587b21e20b7fab14b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260929 11:25:29.484189 19854 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8ed8cdab374587b21e20b7fab14b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36390
I20260929 11:25:29.484894 20228 tablet_replica.cc:330] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: stopping tablet replica
I20260929 11:25:29.485502 20228 raft_consensus.cc:2229] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:29.485651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19791
I20260929 11:25:29.486104 20228 raft_consensus.cc:2258] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:29.488402 20228 ts_tablet_manager.cc:1905] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:29.496454 20230 tablet_replica.cc:330] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: stopping tablet replica
I20260929 11:25:29.497557 20230 raft_consensus.cc:2229] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:25:29.497700 20228 ts_tablet_manager.cc:1918] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:29.497893 20228 log.cc:1196] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/fa8ed8cdab374587b21e20b7fab14b26
I20260929 11:25:29.498477 20228 ts_tablet_manager.cc:1939] T fa8ed8cdab374587b21e20b7fab14b26 P e92c9cdac40e42c08223b1a63b62222a: Deleting consensus metadata
I20260929 11:25:29.498863 20230 raft_consensus.cc:2258] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:29.500716 19714 catalog_manager.cc:4841] TS e92c9cdac40e42c08223b1a63b62222a (127.0.97.3:45529): tablet fa8ed8cdab374587b21e20b7fab14b26 (table test-table [id=59217351166746d5b6e4813cc265d3c0]) successfully deleted
I20260929 11:25:29.501595 20229 tablet_replica.cc:330] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: stopping tablet replica
I20260929 11:25:29.502061 20229 raft_consensus.cc:2229] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:29.502197 20230 ts_tablet_manager.cc:1905] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:29.502592 20229 raft_consensus.cc:2258] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:29.504863 20229 ts_tablet_manager.cc:1905] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:25:29 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 11:25:29.510119 20230 ts_tablet_manager.cc:1918] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:29.510701 20230 log.cc:1196] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/fa8ed8cdab374587b21e20b7fab14b26
I20260929 11:25:29.511299 20230 ts_tablet_manager.cc:1939] T fa8ed8cdab374587b21e20b7fab14b26 P 5dfc0ca752e8486fbbf92911291aef0c: Deleting consensus metadata
I20260929 11:25:29.512355 20229 ts_tablet_manager.cc:1918] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:29.512544 20229 log.cc:1196] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/fa8ed8cdab374587b21e20b7fab14b26
I20260929 11:25:29.513127 20229 ts_tablet_manager.cc:1939] T fa8ed8cdab374587b21e20b7fab14b26 P aff54d08d12249488faa2f1fa72c3016: Deleting consensus metadata
I20260929 11:25:29.513500 19715 catalog_manager.cc:4841] TS 5dfc0ca752e8486fbbf92911291aef0c (127.0.97.2:42833): tablet fa8ed8cdab374587b21e20b7fab14b26 (table test-table [id=59217351166746d5b6e4813cc265d3c0]) successfully deleted
I20260929 11:25:29.514890 19716 catalog_manager.cc:4841] TS aff54d08d12249488faa2f1fa72c3016 (127.0.97.1:33339): tablet fa8ed8cdab374587b21e20b7fab14b26 (table test-table [id=59217351166746d5b6e4813cc265d3c0]) successfully deleted
I20260929 11:25:29.516783 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.487033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.0.97.1:33339)
0929 11:25:29.487223 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.487231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.497144 (+ 9913us) server_negotiation.cc:407] Connection header received
0929 11:25:29.497683 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.497692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.497783 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.497867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:29.500036 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.501253 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.503087 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.503728 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.503846 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.509389 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:29.509406 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:29.509423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:29.509470 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:29.513762 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.514640 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.514649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.514653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.514810 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.515237 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.515243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.515246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.515591 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:29.515915 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.516214 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.516427 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:25:29.517414 19914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:29.632443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19791
I20260929 11:25:29.649747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19925
Sep 29 11:25:29 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 11:25:29.665807 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.650777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.97.2:42833)
0929 11:25:29.650967 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.650975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.650995 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:25:29.651066 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.651071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.651141 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.651261 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:29.652463 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.653336 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.654623 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.655075 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.655206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.659389 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:29.659404 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:29.659456 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:29.659494 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:29.662765 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.663486 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.663494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.663497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.663663 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.664032 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.664037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.664045 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.664395 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:29.664851 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.665205 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.665422 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:25:29.666640 20048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:29.779492 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19925
I20260929 11:25:29.798913 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20058
Sep 29 11:25:29 dist-test-slave-srst krb5kdc[19685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 11:25:29.813752 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:29.799675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.0.97.3:45529)
0929 11:25:29.799908 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:29.799914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:29.799935 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:25:29.800083 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:29.800086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:29.800150 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:29.800241 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:29.801590 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.802301 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.803452 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:29.803819 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:29.803906 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:29.808187 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:29.808207 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:29.808228 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:29.808268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:29.810939 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.811587 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.811594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.811596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.811715 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:29.812088 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:29.812091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:29.812093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:29.812425 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:29.812716 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:29.813126 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 11:25:29.813364 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 11:25:29.814429 20181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:29.901592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20058
I20260929 11:25:29.918502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 19698
I20260929 11:25:29.919576 19760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:30.031499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 19698
2026-09-29T11:25:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T11:25:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:30Z Disabled control of system clock
I20260929 11:25:30.106812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40813
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:25:30.425410 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:30.425664 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:30.425758 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:30.432667 20239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:30.432765 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:30.432807 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:30.432865 20239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:30.432909 20239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:30.441826 20239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:30.443015 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:30.444725 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:30.456306 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:30.456243 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:30.456209 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:30.457249 20239 server_base.cc:1034] running on GCE node
I20260929 11:25:30.457983 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:30.459306 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:30.460541 20239 hybrid_clock.cc:648] HybridClock initialized: now 1790681130460499 us; error 33 us; skew 500 ppm
I20260929 11:25:30.460965 20239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:30.463070 20239 webserver.cc:460] Webserver started at http://127.0.97.62:43441/ using document root <none> and password file <none>
I20260929 11:25:30.463704 20239 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:30.463809 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:30.464123 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:30.466908 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "0da765432ee94e35b5fde96318f065cd"
format_stamp: "Formatted at 2026-09-29 11:25:30 on dist-test-slave-srst"
server_key: "57a18795815491833d7c9724b2cf8a48"
server_key_iv: "f31bd275d1b645ce1a5b43c3bc8917c1"
server_key_version: "encryptionkey@0"
I20260929 11:25:30.467630 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "0da765432ee94e35b5fde96318f065cd"
format_stamp: "Formatted at 2026-09-29 11:25:30 on dist-test-slave-srst"
server_key: "57a18795815491833d7c9724b2cf8a48"
server_key_iv: "f31bd275d1b645ce1a5b43c3bc8917c1"
server_key_version: "encryptionkey@0"
I20260929 11:25:30.472458 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 11:25:30.475741 20253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:30.477078 20239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:30.477246 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "0da765432ee94e35b5fde96318f065cd"
format_stamp: "Formatted at 2026-09-29 11:25:30 on dist-test-slave-srst"
server_key: "57a18795815491833d7c9724b2cf8a48"
server_key_iv: "f31bd275d1b645ce1a5b43c3bc8917c1"
server_key_version: "encryptionkey@0"
I20260929 11:25:30.477455 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:30.506367 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:30.507192 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:30.507474 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:30.525264 20239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40813
I20260929 11:25:30.525296 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40813 every 8 connection(s)
I20260929 11:25:30.526480 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:30.529248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20239
I20260929 11:25:30.529414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:30.530048 388 external_mini_cluster.cc:1428] Setting key 7d8badbfab7ebba91756bd0e98e5a062
I20260929 11:25:30.530311 20305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:30.543815 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:30.531894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.0.97.62:40813)
0929 11:25:30.532861 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:30.532872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:30.532912 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:25:30.534976 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:30.534997 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:30.535445 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:30.535765 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:30.536320 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:30.537693 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:30.539509 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:30.540141 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:30.540284 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:30.541554 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:30.541561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:30.541573 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:30.542119 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:30.542195 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:30.542215 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:30.542300 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:30.542513 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 11:25:30.542883 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":343,"threads_started":1}
I20260929 11:25:30.544000 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd: Bootstrap starting.
I20260929 11:25:30.547070 20305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:30.548039 20305 log.cc:824] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:30.550658 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd: No bootstrap required, opened a new log
I20260929 11:25:30.557583 20305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da765432ee94e35b5fde96318f065cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40813 } }
I20260929 11:25:30.557936 20305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:30.558063 20305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da765432ee94e35b5fde96318f065cd, State: Initialized, Role: FOLLOWER
I20260929 11:25:30.558645 20305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da765432ee94e35b5fde96318f065cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40813 } }
I20260929 11:25:30.558874 20305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:30.559001 20305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:30.559180 20305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:30.561324 20305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da765432ee94e35b5fde96318f065cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40813 } }
I20260929 11:25:30.561801 20305 leader_election.cc:302] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da765432ee94e35b5fde96318f065cd; no voters:
I20260929 11:25:30.562398 20305 leader_election.cc:288] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:30.562494 20310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:30.563205 20310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [term 1 LEADER]: Becoming Leader. State: Replica: 0da765432ee94e35b5fde96318f065cd, State: Running, Role: LEADER
I20260929 11:25:30.563781 20310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da765432ee94e35b5fde96318f065cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40813 } }
I20260929 11:25:30.564393 20305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:30.566375 20312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da765432ee94e35b5fde96318f065cd. Latest consensus state: current_term: 1 leader_uuid: "0da765432ee94e35b5fde96318f065cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da765432ee94e35b5fde96318f065cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40813 } } }
I20260929 11:25:30.566720 20312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:30.568300 20315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:30.568521 20311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0da765432ee94e35b5fde96318f065cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da765432ee94e35b5fde96318f065cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40813 } } }
I20260929 11:25:30.569137 20311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:30.573770 20315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:30.582737 20315 catalog_manager.cc:1347] Generated new cluster ID: 008c93058fbb4e9e857ff1a4570a4b23
I20260929 11:25:30.582839 20315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:30.594051 20315 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:30.595834 20315 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:30.601603 20315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd: Generated new TSK 0
I20260929 11:25:30.602566 20315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:30.615080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40813
--builtin_ntp_servers=127.0.97.20:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:25:30.937297 20329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:30.937553 20329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:30.937635 20329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:30.944543 20329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:30.944653 20329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:30.944758 20329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 11:25:30.953609 20329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:30.954851 20329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:30.956632 20329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:30.969424 20337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:30.969584 20334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:30.969514 20335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:30.970425 20329 server_base.cc:1034] running on GCE node
I20260929 11:25:30.971136 20329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:30.972087 20329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:30.973305 20329 hybrid_clock.cc:648] HybridClock initialized: now 1790681130973254 us; error 43 us; skew 500 ppm
I20260929 11:25:30.973621 20329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:30.975750 20329 webserver.cc:460] Webserver started at http://127.0.97.1:36311/ using document root <none> and password file <none>
I20260929 11:25:30.976432 20329 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:30.976548 20329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:30.976938 20329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:30.979653 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data/instance:
uuid: "6d5a22abe44548ff960437e50b9f3cb3"
format_stamp: "Formatted at 2026-09-29 11:25:30 on dist-test-slave-srst"
server_key: "61190090d95fe40f778438eed8203ea9"
server_key_iv: "b92c5a4bd1c12643241a81a2ce056ccd"
server_key_version: "encryptionkey@0"
I20260929 11:25:30.980453 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6d5a22abe44548ff960437e50b9f3cb3"
format_stamp: "Formatted at 2026-09-29 11:25:30 on dist-test-slave-srst"
server_key: "61190090d95fe40f778438eed8203ea9"
server_key_iv: "b92c5a4bd1c12643241a81a2ce056ccd"
server_key_version: "encryptionkey@0"
I20260929 11:25:30.985670 20329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:25:30.989164 20343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:30.990411 20329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:25:30.990588 20329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal
uuid: "6d5a22abe44548ff960437e50b9f3cb3"
format_stamp: "Formatted at 2026-09-29 11:25:30 on dist-test-slave-srst"
server_key: "61190090d95fe40f778438eed8203ea9"
server_key_iv: "b92c5a4bd1c12643241a81a2ce056ccd"
server_key_version: "encryptionkey@0"
I20260929 11:25:30.990806 20329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:31.001411 20329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:31.002154 20329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:31.002414 20329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:31.003278 20329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:31.004705 20329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:31.004838 20329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.004957 20329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:31.005064 20329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.038118 20329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37723
I20260929 11:25:31.038167 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37723 every 8 connection(s)
I20260929 11:25:31.039443 20329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/data/info.pb
I20260929 11:25:31.048172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20329
I20260929 11:25:31.048339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal/instance
I20260929 11:25:31.048837 388 external_mini_cluster.cc:1428] Setting key 4b332abaf375ce255dae12c4f20a1483
I20260929 11:25:31.051743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40813
--builtin_ntp_servers=127.0.97.20:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:25:31.053211 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.042145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57589 (local address 127.0.97.62:40813)
0929 11:25:31.042326 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.042330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:31.043654 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 11:25:31.045537 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:31.045543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:31.045633 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:31.045747 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:31.046846 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.047627 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.049498 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.050056 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.050197 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.051701 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:31.051708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:31.051712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:31.051735 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:31.051776 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:31.051794 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:31.051889 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:31.052275 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:25:31.052542 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:25:31.054036 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.042579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40813 (local address 127.0.97.1:57589)
0929 11:25:31.043493 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:31.043531 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:31.045269 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:31.045969 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:31.045978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:31.046073 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:31.046685 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.046702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.047780 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.047790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:31.049304 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.049318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.050266 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.050271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:31.050537 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.051302 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:31.051307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:31.051374 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:31.051476 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:31.051964 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:31.051979 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:31.051989 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:25:31.052179 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 11:25:31.052629 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":334,"threads_started":1}
I20260929 11:25:31.055490 20456 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40813
I20260929 11:25:31.055804 20456 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:31.056697 20456 heartbeater.cc:511] Master 127.0.97.62:40813 requested a full tablet report, sending...
I20260929 11:25:31.058760 20270 ts_manager.cc:194] Registered new tserver with Master: 6d5a22abe44548ff960437e50b9f3cb3 (127.0.97.1:37723)
I20260929 11:25:31.060477 20270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:57589
W20260929 11:25:31.371976 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:31.372262 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:31.372349 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:31.379446 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:31.379541 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:31.379644 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 11:25:31.388790 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:31.390064 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:31.391808 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:31.406131 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:31.406087 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:31.406678 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:31.410037 20459 server_base.cc:1034] running on GCE node
I20260929 11:25:31.410624 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:31.411470 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:31.412680 20459 hybrid_clock.cc:648] HybridClock initialized: now 1790681131412629 us; error 50 us; skew 500 ppm
I20260929 11:25:31.412986 20459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:31.415066 20459 webserver.cc:460] Webserver started at http://127.0.97.2:36573/ using document root <none> and password file <none>
I20260929 11:25:31.415709 20459 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:31.415841 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:31.416162 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:31.418840 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data/instance:
uuid: "d2fce551b0024eabb6f11b0210737af7"
format_stamp: "Formatted at 2026-09-29 11:25:31 on dist-test-slave-srst"
server_key: "5ab41ce92a1966d78746e46a53e11252"
server_key_iv: "a7b178addc92d01030132a631b341b87"
server_key_version: "encryptionkey@0"
I20260929 11:25:31.419617 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d2fce551b0024eabb6f11b0210737af7"
format_stamp: "Formatted at 2026-09-29 11:25:31 on dist-test-slave-srst"
server_key: "5ab41ce92a1966d78746e46a53e11252"
server_key_iv: "a7b178addc92d01030132a631b341b87"
server_key_version: "encryptionkey@0"
I20260929 11:25:31.424276 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 11:25:31.427500 20473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.428741 20459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260929 11:25:31.428895 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal
uuid: "d2fce551b0024eabb6f11b0210737af7"
format_stamp: "Formatted at 2026-09-29 11:25:31 on dist-test-slave-srst"
server_key: "5ab41ce92a1966d78746e46a53e11252"
server_key_iv: "a7b178addc92d01030132a631b341b87"
server_key_version: "encryptionkey@0"
I20260929 11:25:31.429107 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:31.446122 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:31.446859 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:31.447130 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:31.447996 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:31.449432 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:31.449544 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.449662 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:31.449743 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.481612 20459 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45747
I20260929 11:25:31.481696 20585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45747 every 8 connection(s)
I20260929 11:25:31.483021 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/data/info.pb
I20260929 11:25:31.487363 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20459
I20260929 11:25:31.487555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal/instance
I20260929 11:25:31.488107 388 external_mini_cluster.cc:1428] Setting key 709e36c300334cfdad6cce4079cb3878
I20260929 11:25:31.491060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40813
--builtin_ntp_servers=127.0.97.20:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:25:31.496711 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.485590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56211 (local address 127.0.97.62:40813)
0929 11:25:31.485771 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.485776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:31.487373 (+ 1597us) server_negotiation.cc:407] Connection header received
0929 11:25:31.489821 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:31.489826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:31.489921 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:31.490025 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:31.491253 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.491985 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.493803 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.494280 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.494418 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.495906 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:31.495911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:31.495914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:31.495933 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:31.495975 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:31.495986 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:31.496079 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:31.496294 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 11:25:31.496392 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:25:31.497778 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.486154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40813 (local address 127.0.97.2:56211)
0929 11:25:31.487179 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:31.487241 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:31.489518 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:31.490309 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:31.490321 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:31.490432 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:31.491034 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.491054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.492141 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.492146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:31.493669 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.493681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.494517 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.494524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:31.494798 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.495563 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:31.495566 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:31.495608 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:31.495694 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:31.496079 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:31.496092 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:31.496103 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:25:31.496286 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 11:25:31.496744 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":278,"threads_started":1}
I20260929 11:25:31.499156 20586 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40813
I20260929 11:25:31.499449 20586 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:31.500173 20586 heartbeater.cc:511] Master 127.0.97.62:40813 requested a full tablet report, sending...
I20260929 11:25:31.501626 20270 ts_manager.cc:194] Registered new tserver with Master: d2fce551b0024eabb6f11b0210737af7 (127.0.97.2:45747)
I20260929 11:25:31.502821 20270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:56211
W20260929 11:25:31.817564 20589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:31.817821 20589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:31.817906 20589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:31.824914 20589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:31.825006 20589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:31.825115 20589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 11:25:31.833936 20589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:31.835140 20589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:31.836804 20589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:31.849270 20594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:31.849295 20595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:31.849561 20597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:31.849893 20589 server_base.cc:1034] running on GCE node
I20260929 11:25:31.850498 20589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:31.851363 20589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:31.852626 20589 hybrid_clock.cc:648] HybridClock initialized: now 1790681131852535 us; error 51 us; skew 500 ppm
I20260929 11:25:31.852934 20589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:25:31.855099 20589 webserver.cc:460] Webserver started at http://127.0.97.3:44657/ using document root <none> and password file <none>
I20260929 11:25:31.855751 20589 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:31.855880 20589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:31.856192 20589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:31.858938 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data/instance:
uuid: "d4bb48bbd70e458787e9fd1c04b88615"
format_stamp: "Formatted at 2026-09-29 11:25:31 on dist-test-slave-srst"
server_key: "6af1e4592449662fd62f2728168b4416"
server_key_iv: "baae040580f82a0f6b12a6d1b4748c0f"
server_key_version: "encryptionkey@0"
I20260929 11:25:31.859704 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d4bb48bbd70e458787e9fd1c04b88615"
format_stamp: "Formatted at 2026-09-29 11:25:31 on dist-test-slave-srst"
server_key: "6af1e4592449662fd62f2728168b4416"
server_key_iv: "baae040580f82a0f6b12a6d1b4748c0f"
server_key_version: "encryptionkey@0"
I20260929 11:25:31.864722 20589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:25:31.867987 20603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.869306 20589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:31.869454 20589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal
uuid: "d4bb48bbd70e458787e9fd1c04b88615"
format_stamp: "Formatted at 2026-09-29 11:25:31 on dist-test-slave-srst"
server_key: "6af1e4592449662fd62f2728168b4416"
server_key_iv: "baae040580f82a0f6b12a6d1b4748c0f"
server_key_version: "encryptionkey@0"
I20260929 11:25:31.869665 20589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:31.895963 20589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:31.896783 20589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:31.897063 20589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:31.898015 20589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:25:31.899478 20589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:25:31.899607 20589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.899725 20589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:25:31.899807 20589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:31.931733 20589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41385
I20260929 11:25:31.931833 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41385 every 8 connection(s)
I20260929 11:25:31.933132 20589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/data/info.pb
I20260929 11:25:31.935300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20589
I20260929 11:25:31.935500 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal/instance
I20260929 11:25:31.936004 388 external_mini_cluster.cc:1428] Setting key 40dbce730e634c05fc050d023ca16e3c
I20260929 11:25:31.945201 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.935650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36575 (local address 127.0.97.62:40813)
0929 11:25:31.935958 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.935965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:31.937462 (+ 1497us) server_negotiation.cc:407] Connection header received
0929 11:25:31.939304 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:31.939308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:31.939371 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:31.939451 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:31.940428 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.941309 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.942906 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.943295 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.943395 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.944518 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:31.944523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:31.944526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:31.944554 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:31.944647 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:31.944667 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:31.944733 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:31.944907 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 11:25:31.944980 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 11:25:31.946081 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.936228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40813 (local address 127.0.97.3:36575)
0929 11:25:31.937300 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:31.937350 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:31.939061 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:31.939642 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:31.939649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:31.939735 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:25:31.940283 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.940303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.941438 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.941442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:31.942638 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.942649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.943470 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.943474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:31.943659 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.944251 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:25:31.944253 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:25:31.944293 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:25:31.944361 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:25:31.944733 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:25:31.944740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:25:31.944745 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:25:31.944880 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 11:25:31.945132 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":377,"threads_started":1}
I20260929 11:25:31.947474 20716 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40813
I20260929 11:25:31.947736 20716 heartbeater.cc:464] Registering TS with master...
I20260929 11:25:31.948416 20716 heartbeater.cc:511] Master 127.0.97.62:40813 requested a full tablet report, sending...
I20260929 11:25:31.949849 20270 ts_manager.cc:194] Registered new tserver with Master: d4bb48bbd70e458787e9fd1c04b88615 (127.0.97.3:41385)
I20260929 11:25:31.950744 20270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36575
I20260929 11:25:31.950927 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:25:31.960875 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.954991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.0.97.62:40813)
0929 11:25:31.955142 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.955161 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:31.955577 (+ 416us) server_negotiation.cc:407] Connection header received
0929 11:25:31.955766 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:31.955771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:31.955871 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:31.955970 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:31.956460 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.957228 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.958576 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:31.958995 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:31.959123 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:31.960114 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:31.960124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:31.960127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:31.960144 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:31.960172 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:31.960183 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:31.960269 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:31.960411 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 11:25:31.960494 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 11:25:31.964248 20270 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:25:31.971113 20270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:25:32.008137 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.982526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37723 (local address 127.0.0.1:35752)
0929 11:25:31.983280 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:31.983318 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:31.983479 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:31.998898 (+ 15419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:31.998904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:31.998967 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:31.999554 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:31.999576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.002582 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.002591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.005220 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.005235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.007127 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.007134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.007353 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.007410 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 11:25:32.007520 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:25:32.007698 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":79,"spinlock_wait_cycles":7424,"thread_start_us":372,"threads_started":1}
I20260929 11:25:32.008860 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.984205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.0.97.2:45747)
0929 11:25:31.996572 (+ 12367us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.996584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.000033 (+ 3449us) server_negotiation.cc:407] Connection header received
0929 11:25:32.000345 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.000355 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.000585 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.000796 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:32.001576 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.003076 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.006145 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.007427 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.007566 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.007789 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.008334 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.008502 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12147,"thread_start_us":12388,"threads_started":1}
I20260929 11:25:32.008919 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.983993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45747 (local address 127.0.0.1:44082)
0929 11:25:31.999901 (+ 15908us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:31.999937 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:32.000142 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:32.001064 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:32.001068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:32.001090 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:32.001420 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.001431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.003255 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.003262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.005925 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.005937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.007897 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.007901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.008026 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.008048 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:25:32.008161 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 11:25:32.008503 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15776,"mutex_wait_us":67,"thread_start_us":12985,"threads_started":1}
I20260929 11:25:32.009470 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.982783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.0.97.1:37723)
0929 11:25:31.997926 (+ 15143us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.997936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:31.997967 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:25:31.998078 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:31.998087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:31.998346 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:31.999259 (+ 913us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:32.000468 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.002409 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.005454 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.006910 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.008640 (+ 1730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.008860 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.008937 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.009108 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14950,"spinlock_wait_cycles":22784,"thread_start_us":14919,"threads_started":1}
I20260929 11:25:32.009754 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.982630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41385 (local address 127.0.0.1:55682)
0929 11:25:31.997229 (+ 14599us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:31.997255 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:31.997393 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:31.998737 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:31.998750 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:31.998819 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:32.000258 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.000272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.004377 (+ 4105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.004382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.006686 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.006699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.008893 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.008900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.009088 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.009119 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:25:32.009215 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:25:32.009343 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14469,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":16966,"threads_started":1}
I20260929 11:25:32.010010 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:31.983164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.0.97.3:41385)
0929 11:25:31.997936 (+ 14772us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:31.997945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:31.997972 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:25:31.998112 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:31.998119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:31.998378 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.000946 (+ 2568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:32.001239 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.004256 (+ 3017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.006915 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.008336 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.008462 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.008760 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.009374 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.009627 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14534,"thread_start_us":14325,"threads_started":1}
I20260929 11:25:32.013100 20521 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1dc40fed0142a5ac066a94d1730a63 (DEFAULT_TABLE table=test-table [id=378f4297e0e64d2aa546ec22352b6e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:32.013142 20391 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1dc40fed0142a5ac066a94d1730a63 (DEFAULT_TABLE table=test-table [id=378f4297e0e64d2aa546ec22352b6e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:32.013142 20651 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1dc40fed0142a5ac066a94d1730a63 (DEFAULT_TABLE table=test-table [id=378f4297e0e64d2aa546ec22352b6e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:25:32.014320 20521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1dc40fed0142a5ac066a94d1730a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:32.014446 20651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1dc40fed0142a5ac066a94d1730a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:32.014443 20391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1dc40fed0142a5ac066a94d1730a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:32.025280 20734 tablet_bootstrap.cc:492] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Bootstrap starting.
I20260929 11:25:32.028683 20735 tablet_bootstrap.cc:492] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Bootstrap starting.
I20260929 11:25:32.029649 20734 tablet_bootstrap.cc:654] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:32.030375 20736 tablet_bootstrap.cc:492] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Bootstrap starting.
I20260929 11:25:32.031095 20734 log.cc:824] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:32.032379 20735 tablet_bootstrap.cc:654] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:32.033581 20736 tablet_bootstrap.cc:654] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:32.033722 20735 log.cc:824] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:32.034459 20734 tablet_bootstrap.cc:492] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: No bootstrap required, opened a new log
I20260929 11:25:32.034657 20736 log.cc:824] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:32.034739 20734 ts_tablet_manager.cc:1397] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 11:25:32.036747 20735 tablet_bootstrap.cc:492] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: No bootstrap required, opened a new log
I20260929 11:25:32.037029 20735 ts_tablet_manager.cc:1397] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:25:32.037238 20736 tablet_bootstrap.cc:492] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: No bootstrap required, opened a new log
I20260929 11:25:32.037518 20736 ts_tablet_manager.cc:1397] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 11:25:32.046164 20736 raft_consensus.cc:348] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.046169 20734 raft_consensus.cc:348] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.046578 20736 raft_consensus.cc:374] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:32.046607 20734 raft_consensus.cc:374] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:32.046666 20736 raft_consensus.cc:729] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bb48bbd70e458787e9fd1c04b88615, State: Initialized, Role: FOLLOWER
I20260929 11:25:32.046713 20734 raft_consensus.cc:729] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fce551b0024eabb6f11b0210737af7, State: Initialized, Role: FOLLOWER
I20260929 11:25:32.047276 20736 consensus_queue.cc:260] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.047376 20734 consensus_queue.cc:260] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.049986 20735 raft_consensus.cc:348] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.050578 20735 raft_consensus.cc:374] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:32.050715 20735 raft_consensus.cc:729] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5a22abe44548ff960437e50b9f3cb3, State: Initialized, Role: FOLLOWER
I20260929 11:25:32.051587 20735 consensus_queue.cc:260] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.055150 20716 heartbeater.cc:503] Master 127.0.97.62:40813 was elected leader, sending a full tablet report...
I20260929 11:25:32.055959 20586 heartbeater.cc:503] Master 127.0.97.62:40813 was elected leader, sending a full tablet report...
I20260929 11:25:32.056984 20734 ts_tablet_manager.cc:1428] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Time spent starting tablet: real 0.022s user 0.017s sys 0.003s
I20260929 11:25:32.058769 20736 ts_tablet_manager.cc:1428] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260929 11:25:32.059113 20456 heartbeater.cc:503] Master 127.0.97.62:40813 was elected leader, sending a full tablet report...
I20260929 11:25:32.059355 20735 ts_tablet_manager.cc:1428] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Time spent starting tablet: real 0.022s user 0.010s sys 0.007s
W20260929 11:25:32.184835 20717 tablet.cc:2369] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:32.234992 20587 tablet.cc:2369] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:25:32.292140 20457 tablet.cc:2369] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:25:32.438669 20741 raft_consensus.cc:482] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:25:32.438989 20741 raft_consensus.cc:504] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.440722 20741 leader_election.cc:288] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d5a22abe44548ff960437e50b9f3cb3 (127.0.97.1:37723), d4bb48bbd70e458787e9fd1c04b88615 (127.0.97.3:41385)
I20260929 11:25:32.448145 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.441229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51551 (local address 127.0.97.1:37723)
0929 11:25:32.441434 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:32.441438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.442279 (+ 841us) server_negotiation.cc:407] Connection header received
0929 11:25:32.442468 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.442473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.442567 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.442666 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:32.443430 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.444471 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.446472 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.447244 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.447377 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.447479 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.447770 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.447889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:25:32.448247 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.441119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37321 (local address 127.0.97.3:41385)
0929 11:25:32.441327 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:32.441334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.442295 (+ 961us) server_negotiation.cc:407] Connection header received
0929 11:25:32.442460 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.442465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.442549 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.442667 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:25:32.443391 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.444506 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.446448 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.447379 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.447504 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.447611 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.447887 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.448010 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:25:32.448271 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.441216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37723 (local address 127.0.97.2:51551)
0929 11:25:32.442128 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:32.442148 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:32.442319 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:32.442694 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:32.442699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:32.442735 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:32.443290 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.443302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.444643 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.444648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.446261 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.446273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.447428 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.447434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.447615 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.447639 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:25:32.447729 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:25:32.447846 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":115,"spinlock_wait_cycles":3328,"thread_start_us":456,"threads_started":1}
I20260929 11:25:32.448408 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.441216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41385 (local address 127.0.97.2:37321)
0929 11:25:32.442197 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 11:25:32.442214 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:25:32.442319 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:25:32.442694 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:25:32.442699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:25:32.442729 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:25:32.443191 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.443211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.444685 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.444688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.446255 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:25:32.446267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.447564 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.447571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:25:32.447730 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.447752 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:25:32.447848 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:25:32.447958 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":159,"spinlock_wait_cycles":53248,"thread_start_us":524,"threads_started":1}
I20260929 11:25:32.449345 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1dc40fed0142a5ac066a94d1730a63" candidate_uuid: "d2fce551b0024eabb6f11b0210737af7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bb48bbd70e458787e9fd1c04b88615" is_pre_election: true
I20260929 11:25:32.449782 20671 raft_consensus.cc:2454] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2fce551b0024eabb6f11b0210737af7 in term 0.
I20260929 11:25:32.449738 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1dc40fed0142a5ac066a94d1730a63" candidate_uuid: "d2fce551b0024eabb6f11b0210737af7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5a22abe44548ff960437e50b9f3cb3" is_pre_election: true
I20260929 11:25:32.450269 20411 raft_consensus.cc:2454] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2fce551b0024eabb6f11b0210737af7 in term 0.
I20260929 11:25:32.450373 20475 leader_election.cc:302] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2fce551b0024eabb6f11b0210737af7, d4bb48bbd70e458787e9fd1c04b88615; no voters:
I20260929 11:25:32.450794 20741 raft_consensus.cc:2781] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:25:32.450927 20741 raft_consensus.cc:482] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:25:32.451025 20741 raft_consensus.cc:3037] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:32.453704 20741 raft_consensus.cc:504] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.454313 20741 leader_election.cc:288] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [CANDIDATE]: Term 1 election: Requested vote from peers 6d5a22abe44548ff960437e50b9f3cb3 (127.0.97.1:37723), d4bb48bbd70e458787e9fd1c04b88615 (127.0.97.3:41385)
I20260929 11:25:32.454742 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1dc40fed0142a5ac066a94d1730a63" candidate_uuid: "d2fce551b0024eabb6f11b0210737af7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bb48bbd70e458787e9fd1c04b88615"
I20260929 11:25:32.454732 20411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1dc40fed0142a5ac066a94d1730a63" candidate_uuid: "d2fce551b0024eabb6f11b0210737af7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5a22abe44548ff960437e50b9f3cb3"
I20260929 11:25:32.454936 20671 raft_consensus.cc:3037] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:32.454946 20411 raft_consensus.cc:3037] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:32.458089 20671 raft_consensus.cc:2454] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2fce551b0024eabb6f11b0210737af7 in term 1.
I20260929 11:25:32.458089 20411 raft_consensus.cc:2454] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2fce551b0024eabb6f11b0210737af7 in term 1.
I20260929 11:25:32.458465 20475 leader_election.cc:302] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d5a22abe44548ff960437e50b9f3cb3, d2fce551b0024eabb6f11b0210737af7, d4bb48bbd70e458787e9fd1c04b88615; no voters:
I20260929 11:25:32.458778 20741 raft_consensus.cc:2781] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:32.459198 20741 raft_consensus.cc:686] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 1 LEADER]: Becoming Leader. State: Replica: d2fce551b0024eabb6f11b0210737af7, State: Running, Role: LEADER
I20260929 11:25:32.459641 20741 consensus_queue.cc:237] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } }
I20260929 11:25:32.464288 20268 catalog_manager.cc:5495] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2fce551b0024eabb6f11b0210737af7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d2fce551b0024eabb6f11b0210737af7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2fce551b0024eabb6f11b0210737af7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45747 } health_report { overall_health: HEALTHY } } }
I20260929 11:25:32.577865 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.571920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.0.97.2:45747)
0929 11:25:32.572691 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:32.572699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.572722 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:25:32.572824 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.572828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.572916 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.573081 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:25:32.573596 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.574268 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.575679 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.576103 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.576238 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.576751 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:25:32.577273 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:25:32.577448 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.577507 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.577583 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":519,"threads_started":1}
I20260929 11:25:32.585839 20671 raft_consensus.cc:1261] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 1 FOLLOWER]: Refusing update from remote peer d2fce551b0024eabb6f11b0210737af7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:32.585868 20411 raft_consensus.cc:1261] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 1 FOLLOWER]: Refusing update from remote peer d2fce551b0024eabb6f11b0210737af7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:25:32.586918 20745 consensus_queue.cc:1035] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4bb48bbd70e458787e9fd1c04b88615" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:32.587237 20741 consensus_queue.cc:1035] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5a22abe44548ff960437e50b9f3cb3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:25:32.611740 20268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:32.612054 20268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:25:32.615446 20268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0da765432ee94e35b5fde96318f065cd: Sending DeleteTablet for 3 replicas of tablet 7d1dc40fed0142a5ac066a94d1730a63
I20260929 11:25:32.616657 20651 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1dc40fed0142a5ac066a94d1730a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:32 UTC) from {username='slave'} at 127.0.0.1:55682
I20260929 11:25:32.616681 20521 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1dc40fed0142a5ac066a94d1730a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:32 UTC) from {username='slave'} at 127.0.0.1:44082
I20260929 11:25:32.616649 20391 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1dc40fed0142a5ac066a94d1730a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:25:32 UTC) from {username='slave'} at 127.0.0.1:35752
I20260929 11:25:32.617492 20762 tablet_replica.cc:330] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: stopping tablet replica
I20260929 11:25:32.618018 20762 raft_consensus.cc:2229] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:32.618702 20762 raft_consensus.cc:2258] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:32.621007 20762 ts_tablet_manager.cc:1905] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:32.621807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20329
I20260929 11:25:32.629860 20762 ts_tablet_manager.cc:1918] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:32.630138 20762 log.cc:1196] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-0/wal/wals/7d1dc40fed0142a5ac066a94d1730a63
I20260929 11:25:32.630749 20762 ts_tablet_manager.cc:1939] T 7d1dc40fed0142a5ac066a94d1730a63 P 6d5a22abe44548ff960437e50b9f3cb3: Deleting consensus metadata
I20260929 11:25:32.633126 20257 catalog_manager.cc:4841] TS 6d5a22abe44548ff960437e50b9f3cb3 (127.0.97.1:37723): tablet 7d1dc40fed0142a5ac066a94d1730a63 (table test-table [id=378f4297e0e64d2aa546ec22352b6e39]) successfully deleted
I20260929 11:25:32.635993 20763 tablet_replica.cc:330] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: stopping tablet replica
I20260929 11:25:32.636507 20763 raft_consensus.cc:2229] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:25:32.636741 20764 tablet_replica.cc:330] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: stopping tablet replica
I20260929 11:25:32.637063 20763 raft_consensus.cc:2258] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:32.637142 20764 raft_consensus.cc:2229] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:25:32.637868 20764 raft_consensus.cc:2258] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:25:32.639146 20763 ts_tablet_manager.cc:1905] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:32.639739 20764 ts_tablet_manager.cc:1905] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:25:32.641557 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.623738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.0.97.1:37723)
0929 11:25:32.623912 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:32.623919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.626316 (+ 2397us) server_negotiation.cc:407] Connection header received
0929 11:25:32.627207 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.627215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.627322 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.627413 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:32.628566 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.629713 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.633431 (+ 3718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.633973 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.634046 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.639745 (+ 5699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:32.639757 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:32.639774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:32.640516 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:32.640576 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:32.640639 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:32.640721 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.641077 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.641190 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:25:32.642876 20451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:32.646018 20763 ts_tablet_manager.cc:1918] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:32.646245 20763 log.cc:1196] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-2/wal/wals/7d1dc40fed0142a5ac066a94d1730a63
I20260929 11:25:32.646793 20763 ts_tablet_manager.cc:1939] T 7d1dc40fed0142a5ac066a94d1730a63 P d4bb48bbd70e458787e9fd1c04b88615: Deleting consensus metadata
I20260929 11:25:32.647516 20764 ts_tablet_manager.cc:1918] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:25:32.647715 20764 log.cc:1196] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Deleting WAL directory at /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790680821855275-388-0/minicluster-data/ts-1/wal/wals/7d1dc40fed0142a5ac066a94d1730a63
I20260929 11:25:32.648283 20764 ts_tablet_manager.cc:1939] T 7d1dc40fed0142a5ac066a94d1730a63 P d2fce551b0024eabb6f11b0210737af7: Deleting consensus metadata
I20260929 11:25:32.648772 20255 catalog_manager.cc:4841] TS d4bb48bbd70e458787e9fd1c04b88615 (127.0.97.3:41385): tablet 7d1dc40fed0142a5ac066a94d1730a63 (table test-table [id=378f4297e0e64d2aa546ec22352b6e39]) successfully deleted
I20260929 11:25:32.649981 20254 catalog_manager.cc:4841] TS d2fce551b0024eabb6f11b0210737af7 (127.0.97.2:45747): tablet 7d1dc40fed0142a5ac066a94d1730a63 (table test-table [id=378f4297e0e64d2aa546ec22352b6e39]) successfully deleted
I20260929 11:25:32.739085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20329
I20260929 11:25:32.756408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20459
I20260929 11:25:32.763505 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.757286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.0.97.2:45747)
0929 11:25:32.757458 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:32.757466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.757486 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:25:32.757685 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.757691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.757804 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.757926 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:32.758363 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.759168 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.760408 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.760886 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.761012 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.762057 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:32.762065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:32.762078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:32.762710 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:32.762785 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:32.762803 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:32.762921 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.763059 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.763155 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:25:32.764134 20581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:32.874271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20459
I20260929 11:25:32.892884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20589
I20260929 11:25:32.899355 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:32.893618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.0.97.3:41385)
0929 11:25:32.893798 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:32.893802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:32.893924 (+ 122us) server_negotiation.cc:407] Connection header received
0929 11:25:32.894073 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:32.894077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:32.894170 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:32.894257 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:32.894657 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.895392 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.896707 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:32.897112 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:32.897214 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:32.898192 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:32.898200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:32.898214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:32.898770 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:32.898815 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:32.898834 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:32.898929 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:32.899039 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 11:25:32.899126 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:25:32.900050 20711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:32.987361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20589
I20260929 11:25:33.004240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20239
I20260929 11:25:33.005333 20300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:33.116709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20239
2026-09-29T11:25:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3083 ms)
[----------] 6 tests from EncryptionPolicyTest (21631 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:33 dist-test-slave-srst krb5kdc[20772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:33 dist-test-slave-srst krb5kdc[20772](info): set up 2 sockets
Sep 29 11:25:33 dist-test-slave-srst krb5kdc[20772](info): commencing operation
krb5kdc: starting...
W20260929 11:25:34.214856 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:34 dist-test-slave-srst krb5kdc[20772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:34.463920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:34.781821 20785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:34.782104 20785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:34.782285 20785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:34.789424 20785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:34.789553 20785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:34.789638 20785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:34.789740 20785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:34.789808 20785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:34.799578 20785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:34.800848 20785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:34.802114 20785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:34.813791 20791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:34.814040 20793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:34.814168 20790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:34.814684 20785 server_base.cc:1034] running on GCE node
I20260929 11:25:34.815395 20785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:34.816756 20785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:34.817982 20785 hybrid_clock.cc:648] HybridClock initialized: now 1790681134817946 us; error 46 us; skew 500 ppm
I20260929 11:25:34.818320 20785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:34 dist-test-slave-srst krb5kdc[20772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:34.821300 20785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:34.822706 20785 webserver.cc:460] Webserver started at http://127.0.0.1:41329/ using document root <none> and password file <none>
I20260929 11:25:34.823360 20785 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:34.823498 20785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:34.823805 20785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:34.826674 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce"
format_stamp: "Formatted at 2026-09-29 11:25:34 on dist-test-slave-srst"
server_key: "d301993a905cc210d52da96b73d332f1"
server_key_iv: "511f421a696624bb8aeb32ddd70495a0"
server_key_version: "encryptionkey@0"
I20260929 11:25:34.827412 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce"
format_stamp: "Formatted at 2026-09-29 11:25:34 on dist-test-slave-srst"
server_key: "d301993a905cc210d52da96b73d332f1"
server_key_iv: "511f421a696624bb8aeb32ddd70495a0"
server_key_version: "encryptionkey@0"
I20260929 11:25:34.832261 20785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 11:25:34.835616 20800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:34.836894 20785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:34.837087 20785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce"
format_stamp: "Formatted at 2026-09-29 11:25:34 on dist-test-slave-srst"
server_key: "d301993a905cc210d52da96b73d332f1"
server_key_iv: "511f421a696624bb8aeb32ddd70495a0"
server_key_version: "encryptionkey@0"
I20260929 11:25:34.837277 20785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:34.858204 20785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:34.862084 20785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:34.862358 20785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:34.880400 20785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36099
I20260929 11:25:34.880478 20851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36099 every 8 connection(s)
I20260929 11:25:34.881676 20785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:34.885375 20852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:34.886690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20785
I20260929 11:25:34.886926 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:34.887830 388 external_mini_cluster.cc:1428] Setting key f92bb310ba76e83aff07834159f918db
I20260929 11:25:34.900386 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce: Bootstrap starting.
Sep 29 11:25:34 dist-test-slave-srst krb5kdc[20772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:25:34.904106 20852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:34.905347 20852 log.cc:824] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:34.908263 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce: No bootstrap required, opened a new log
I20260929 11:25:34.910929 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:34.889809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50637 (local address 127.0.0.1:36099)
0929 11:25:34.890565 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:34.890576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:34.890613 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:25:34.892199 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:34.892223 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:34.892643 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:34.893001 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:34.894953 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:34.896198 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:34.898204 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:34.898814 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:34.898958 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:34.903913 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:34.903921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:34.903934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:34.903967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:34.906975 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:34.907734 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:34.907740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:34.907746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:34.907901 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:34.908344 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:34.908351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:34.908359 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:34.908957 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:34.909284 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:34.909714 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 11:25:34.910030 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":361,"threads_started":1}
I20260929 11:25:34.914979 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:34.917127 20852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260929 11:25:34.917493 20852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:34.917620 20852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfa1babed9a4c93aaf2c3d75ca81cce, State: Initialized, Role: FOLLOWER
I20260929 11:25:34.918251 20852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260929 11:25:34.918593 20852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:34.918766 20852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:34.918973 20852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:34.921723 20852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260929 11:25:34.922258 20852 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cfa1babed9a4c93aaf2c3d75ca81cce; no voters:
I20260929 11:25:34.922994 20852 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:34.923091 20862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:34.924865 20862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [term 1 LEADER]: Becoming Leader. State: Replica: 9cfa1babed9a4c93aaf2c3d75ca81cce, State: Running, Role: LEADER
I20260929 11:25:34.925616 20852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:34.925531 20862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } }
I20260929 11:25:34.928686 20863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfa1babed9a4c93aaf2c3d75ca81cce. Latest consensus state: current_term: 1 leader_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } } }
I20260929 11:25:34.929304 20863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:34.929248 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfa1babed9a4c93aaf2c3d75ca81cce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36099 } } }
I20260929 11:25:34.929905 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:34.930735 20871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:34.939081 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:34.919792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39813 (local address 127.0.0.1:36099)
0929 11:25:34.919968 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:34.919975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:34.920378 (+ 403us) server_negotiation.cc:407] Connection header received
0929 11:25:34.920580 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:34.920620 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:34.920688 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:34.920791 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:34.922180 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:34.923009 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:34.924863 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:34.925429 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:34.925532 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:34.928828 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:34.928837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:34.928844 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:34.928893 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:34.934015 (+ 5122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:34.935433 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:34.935443 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:34.935449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:34.935639 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:34.936311 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:34.936319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:34.936323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:34.936823 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:34.937130 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:34.938460 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0929 11:25:34.938648 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:25:34.942625 20871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:34.952301 20871 catalog_manager.cc:1347] Generated new cluster ID: 427048cb7c0149b296450d2554604981
I20260929 11:25:34.952409 20871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:34.982854 20871 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:34.984186 20871 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:34.997421 20871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cfa1babed9a4c93aaf2c3d75ca81cce: Generated new TSK 0
I20260929 11:25:34.998557 20871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:35.013101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20785
I20260929 11:25:35.014323 20847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:35.107097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20785
W20260929 11:25:35.124511 20770 connection.cc:535] client connection to 127.0.0.1:36099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36099): Connection reset by peer (error 104)
2026-09-29T11:25:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:35 dist-test-slave-srst krb5kdc[20884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:35 dist-test-slave-srst krb5kdc[20884](info): set up 2 sockets
Sep 29 11:25:35 dist-test-slave-srst krb5kdc[20884](info): commencing operation
krb5kdc: starting...
W20260929 11:25:36.200950 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:36 dist-test-slave-srst krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:36.443436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:36.767879 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:36.768153 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:36.768242 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:36.775256 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:36.775363 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:36.775420 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:36.775488 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:36.775527 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:36.785101 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:36.786332 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:36.787606 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:36.798938 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:36.799019 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:36.799098 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:36.800166 20897 server_base.cc:1034] running on GCE node
I20260929 11:25:36.800952 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:36.802376 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:36.803567 20897 hybrid_clock.cc:648] HybridClock initialized: now 1790681136803526 us; error 37 us; skew 500 ppm
I20260929 11:25:36.803896 20897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:36 dist-test-slave-srst krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:36.806951 20897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:36.808295 20897 webserver.cc:460] Webserver started at http://127.0.0.1:41007/ using document root <none> and password file <none>
I20260929 11:25:36.809043 20897 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:36.809187 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:36.809505 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:36.812536 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "430a2d3265384db6935b506f7aa6f0ca"
format_stamp: "Formatted at 2026-09-29 11:25:36 on dist-test-slave-srst"
server_key: "2255e0c5ca3379d19dedf151db205ee4"
server_key_iv: "47c4768100d72a98e9d08c1f6315520b"
server_key_version: "encryptionkey@0"
I20260929 11:25:36.813421 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "430a2d3265384db6935b506f7aa6f0ca"
format_stamp: "Formatted at 2026-09-29 11:25:36 on dist-test-slave-srst"
server_key: "2255e0c5ca3379d19dedf151db205ee4"
server_key_iv: "47c4768100d72a98e9d08c1f6315520b"
server_key_version: "encryptionkey@0"
I20260929 11:25:36.818742 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:25:36.822202 20912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:36.823459 20897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:25:36.823632 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "430a2d3265384db6935b506f7aa6f0ca"
format_stamp: "Formatted at 2026-09-29 11:25:36 on dist-test-slave-srst"
server_key: "2255e0c5ca3379d19dedf151db205ee4"
server_key_iv: "47c4768100d72a98e9d08c1f6315520b"
server_key_version: "encryptionkey@0"
I20260929 11:25:36.823843 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:36.840801 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:36.844841 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:36.845120 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:36.863826 20897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36641
I20260929 11:25:36.863860 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36641 every 8 connection(s)
I20260929 11:25:36.865362 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:36.866367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 20897
I20260929 11:25:36.866586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:36.867173 388 external_mini_cluster.cc:1428] Setting key 087fcaefe01953fbb7c7db7bf10a74ce
I20260929 11:25:36.870193 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:36.879918 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:36.869132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60755 (local address 127.0.0.1:36641)
0929 11:25:36.869878 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:36.869890 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:36.869929 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:25:36.871621 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:36.871643 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:36.872022 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:36.872401 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:36.873010 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:36.874272 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:36.875569 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:36.876250 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:36.876451 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:36.877759 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:36.877767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:36.877780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:36.878332 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:36.878404 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:36.878430 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:36.878599 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:36.878900 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 11:25:36.879045 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260929 11:25:36.880956 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca: Bootstrap starting.
I20260929 11:25:36.883508 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:36.884214 20964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:36.885258 20964 log.cc:824] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:36.888391 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca: No bootstrap required, opened a new log
I20260929 11:25:36.894433 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:36.887817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55513 (local address 127.0.0.1:36641)
0929 11:25:36.887991 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:36.887999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:36.888391 (+ 392us) server_negotiation.cc:407] Connection header received
0929 11:25:36.888615 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:36.888620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:36.888698 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:36.888800 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:36.889245 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:36.890358 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:36.891686 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:36.892397 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:36.892528 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:36.893651 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:36.893657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:36.893662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:36.893682 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:36.893744 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:36.893771 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:36.893888 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:36.894037 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 11:25:36.894092 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:25:36.897385 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430a2d3265384db6935b506f7aa6f0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36641 } }
I20260929 11:25:36.897742 20964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:36.897859 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430a2d3265384db6935b506f7aa6f0ca, State: Initialized, Role: FOLLOWER
I20260929 11:25:36.898424 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430a2d3265384db6935b506f7aa6f0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36641 } }
I20260929 11:25:36.898681 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:36.898845 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:36.899030 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:36.901206 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430a2d3265384db6935b506f7aa6f0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36641 } }
I20260929 11:25:36.901688 20964 leader_election.cc:302] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 430a2d3265384db6935b506f7aa6f0ca; no voters:
I20260929 11:25:36.902282 20964 leader_election.cc:288] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:36.902407 20974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:36.903085 20974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [term 1 LEADER]: Becoming Leader. State: Replica: 430a2d3265384db6935b506f7aa6f0ca, State: Running, Role: LEADER
I20260929 11:25:36.903687 20974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430a2d3265384db6935b506f7aa6f0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36641 } }
I20260929 11:25:36.904361 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:36.906121 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "430a2d3265384db6935b506f7aa6f0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430a2d3265384db6935b506f7aa6f0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36641 } } }
I20260929 11:25:36.906409 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:36.907455 20979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:36.907363 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 430a2d3265384db6935b506f7aa6f0ca. Latest consensus state: current_term: 1 leader_uuid: "430a2d3265384db6935b506f7aa6f0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430a2d3265384db6935b506f7aa6f0ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36641 } } }
I20260929 11:25:36.907672 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:36.913064 20979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:36.922578 20979 catalog_manager.cc:1347] Generated new cluster ID: f5a18922f2594ed39eaf66efe1324d02
I20260929 11:25:36.922695 20979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:36.939863 20979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:36.941763 20979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:36.965771 20979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 430a2d3265384db6935b506f7aa6f0ca: Generated new TSK 0
I20260929 11:25:36.966849 20979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:36.981350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20897
W20260929 11:25:36.982406 20959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60755
W20260929 11:25:36.983134 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 20897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:36.983302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 20897
W20260929 11:25:37.000204 20882 connection.cc:535] client connection to 127.0.0.1:36641 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36641): Connection reset by peer (error 104)
2026-09-29T11:25:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:37 dist-test-slave-srst krb5kdc[20995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:37 dist-test-slave-srst krb5kdc[20995](info): set up 2 sockets
Sep 29 11:25:37 dist-test-slave-srst krb5kdc[20995](info): commencing operation
krb5kdc: starting...
W20260929 11:25:38.085076 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:38 dist-test-slave-srst krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:38.322098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:38.644867 21008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:38.645165 21008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:38.645349 21008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:38.652602 21008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:38.652734 21008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:38.652833 21008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:38.652952 21008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:38.653030 21008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:38.662673 21008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:38.663964 21008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:38.665318 21008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:38.677021 21013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:38.677066 21016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:38.676975 21014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:38.677968 21008 server_base.cc:1034] running on GCE node
I20260929 11:25:38.678710 21008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:38.680045 21008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:38.681265 21008 hybrid_clock.cc:648] HybridClock initialized: now 1790681138681231 us; error 44 us; skew 500 ppm
I20260929 11:25:38.681605 21008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:38 dist-test-slave-srst krb5kdc[20995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:38.688066 21008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:38.689518 21008 webserver.cc:460] Webserver started at http://127.0.0.1:36943/ using document root <none> and password file <none>
I20260929 11:25:38.690182 21008 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:38.690321 21008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:38.690621 21008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:38.693643 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "c7fa4013d9534c869039a3dfdca8815f"
format_stamp: "Formatted at 2026-09-29 11:25:38 on dist-test-slave-srst"
server_key: "d8d99c1d401c887ce86ba2aa6a972686"
server_key_iv: "71994d59336104a519c39aedd5cec12a"
server_key_version: "encryptionkey@0"
I20260929 11:25:38.694418 21008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "c7fa4013d9534c869039a3dfdca8815f"
format_stamp: "Formatted at 2026-09-29 11:25:38 on dist-test-slave-srst"
server_key: "d8d99c1d401c887ce86ba2aa6a972686"
server_key_iv: "71994d59336104a519c39aedd5cec12a"
server_key_version: "encryptionkey@0"
I20260929 11:25:38.699558 21008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:38.702967 21023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:38.704252 21008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:25:38.704442 21008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "c7fa4013d9534c869039a3dfdca8815f"
format_stamp: "Formatted at 2026-09-29 11:25:38 on dist-test-slave-srst"
server_key: "d8d99c1d401c887ce86ba2aa6a972686"
server_key_iv: "71994d59336104a519c39aedd5cec12a"
server_key_version: "encryptionkey@0"
I20260929 11:25:38.704677 21008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:38.710126 21008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:38.713999 21008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:38.714282 21008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:38.731925 21008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42217
I20260929 11:25:38.731964 21074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42217 every 8 connection(s)
I20260929 11:25:38.733419 21008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:38.737046 21075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:38.743415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21008
I20260929 11:25:38.743597 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:38.744091 388 external_mini_cluster.cc:1428] Setting key f2f3b6376a36a256c241888040bd0cac
I20260929 11:25:38.748109 21075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f: Bootstrap starting.
I20260929 11:25:38.751591 21075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:38.752538 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:38.745977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33727 (local address 127.0.0.1:42217)
0929 11:25:38.747426 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:38.747438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:38.747477 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:25:38.749277 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:38.749299 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:38.749643 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:38.749952 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:38.750967 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:38.750973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:38.750986 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:38.750999 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:38.751061 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:38.751081 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:38.751211 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:38.751411 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 11:25:38.751619 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":987,"threads_started":1}
I20260929 11:25:38.752710 21075 log.cc:824] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:38.755558 21075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f: No bootstrap required, opened a new log
I20260929 11:25:38.756124 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:38.762862 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:38.760403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57423 (local address 127.0.0.1:42217)
0929 11:25:38.760548 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:38.760552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:38.760995 (+ 443us) server_negotiation.cc:407] Connection header received
0929 11:25:38.761185 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:38.761192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:38.761253 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:38.761362 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:38.762232 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:38.762237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:38.762241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:38.762258 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:38.762308 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:38.762335 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:38.762457 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:38.762546 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 11:25:38.762598 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:25:38.763448 21075 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7fa4013d9534c869039a3dfdca8815f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260929 11:25:38.763890 21075 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:38.763995 21075 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fa4013d9534c869039a3dfdca8815f, State: Initialized, Role: FOLLOWER
I20260929 11:25:38.764586 21075 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7fa4013d9534c869039a3dfdca8815f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260929 11:25:38.764902 21075 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:38.765043 21075 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:38.765209 21075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:38.767234 21075 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7fa4013d9534c869039a3dfdca8815f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260929 11:25:38.767689 21075 leader_election.cc:302] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7fa4013d9534c869039a3dfdca8815f; no voters:
I20260929 11:25:38.768239 21075 leader_election.cc:288] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:38.768369 21085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:38.769721 21085 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [term 1 LEADER]: Becoming Leader. State: Replica: c7fa4013d9534c869039a3dfdca8815f, State: Running, Role: LEADER
I20260929 11:25:38.770197 21075 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:38.770179 21085 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7fa4013d9534c869039a3dfdca8815f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260929 11:25:38.772708 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7fa4013d9534c869039a3dfdca8815f. Latest consensus state: current_term: 1 leader_uuid: "c7fa4013d9534c869039a3dfdca8815f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7fa4013d9534c869039a3dfdca8815f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } } }
I20260929 11:25:38.772897 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7fa4013d9534c869039a3dfdca8815f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7fa4013d9534c869039a3dfdca8815f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } } }
I20260929 11:25:38.773136 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:38.773156 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:38.774071 21093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:38.780045 21093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:38.790308 21093 catalog_manager.cc:1347] Generated new cluster ID: 72d162b42d994508814d2851d6ab69b2
I20260929 11:25:38.790475 21093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:38.817220 21093 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:38.817648 21093 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:38.835772 21093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7fa4013d9534c869039a3dfdca8815f: Generated new TSK 0
I20260929 11:25:38.836869 21093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:38.848798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21008
W20260929 11:25:38.849833 21070 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33727
W20260929 11:25:38.850534 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:38.850665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21008
2026-09-29T11:25:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:38 dist-test-slave-srst krb5kdc[21106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:38 dist-test-slave-srst krb5kdc[21106](info): set up 2 sockets
Sep 29 11:25:38 dist-test-slave-srst krb5kdc[21106](info): commencing operation
krb5kdc: starting...
W20260929 11:25:39.932906 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:40 dist-test-slave-srst krb5kdc[21106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:40.179657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:40.506880 21119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:40.507160 21119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:40.507313 21119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:40.514400 21119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:40.514497 21119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:40.514540 21119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:40.514626 21119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:40.514714 21119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:40.524062 21119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:40.525319 21119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:40.526572 21119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:40.537665 21124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:40.537789 21125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:40.537811 21127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:40.538770 21119 server_base.cc:1034] running on GCE node
I20260929 11:25:40.539482 21119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:40.540902 21119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:40.542115 21119 hybrid_clock.cc:648] HybridClock initialized: now 1790681140542071 us; error 57 us; skew 500 ppm
I20260929 11:25:40.542428 21119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:40 dist-test-slave-srst krb5kdc[21106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:40.545500 21119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:40.546960 21119 webserver.cc:460] Webserver started at http://127.0.0.1:45349/ using document root <none> and password file <none>
I20260929 11:25:40.547616 21119 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:40.547775 21119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:40.548076 21119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:40.551059 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "f60b37a21bee4e279ab70858f756e703"
format_stamp: "Formatted at 2026-09-29 11:25:40 on dist-test-slave-srst"
server_key: "91f153a055fd7c8e2e3f59d273af44c5"
server_key_iv: "112c2428275cc99ce75b062b841c0653"
server_key_version: "encryptionkey@0"
I20260929 11:25:40.551833 21119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "f60b37a21bee4e279ab70858f756e703"
format_stamp: "Formatted at 2026-09-29 11:25:40 on dist-test-slave-srst"
server_key: "91f153a055fd7c8e2e3f59d273af44c5"
server_key_iv: "112c2428275cc99ce75b062b841c0653"
server_key_version: "encryptionkey@0"
I20260929 11:25:40.556926 21119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:25:40.560304 21134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:40.561628 21119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:40.561815 21119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "f60b37a21bee4e279ab70858f756e703"
format_stamp: "Formatted at 2026-09-29 11:25:40 on dist-test-slave-srst"
server_key: "91f153a055fd7c8e2e3f59d273af44c5"
server_key_iv: "112c2428275cc99ce75b062b841c0653"
server_key_version: "encryptionkey@0"
I20260929 11:25:40.562024 21119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:40.574150 21119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:40.578008 21119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:40.578234 21119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:40.596073 21119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41667
I20260929 11:25:40.596113 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41667 every 8 connection(s)
I20260929 11:25:40.597440 21119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:40.601068 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:40.606848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21119
I20260929 11:25:40.607041 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:40.607438 388 external_mini_cluster.cc:1428] Setting key bbdb798a7fd756a4041573f859856eef
I20260929 11:25:40.611726 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703: Bootstrap starting.
I20260929 11:25:40.615525 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:40.616761 21186 log.cc:824] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:40.619812 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703: No bootstrap required, opened a new log
Sep 29 11:25:40 dist-test-slave-srst krb5kdc[21106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:25:40.628691 21186 raft_consensus.cc:348] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60b37a21bee4e279ab70858f756e703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260929 11:25:40.629045 21186 raft_consensus.cc:374] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:40.629144 21186 raft_consensus.cc:729] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60b37a21bee4e279ab70858f756e703, State: Initialized, Role: FOLLOWER
I20260929 11:25:40.629782 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60b37a21bee4e279ab70858f756e703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260929 11:25:40.630031 21186 raft_consensus.cc:388] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:40.630218 21186 raft_consensus.cc:482] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:40.630446 21186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:40.631325 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:40.609257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35705 (local address 127.0.0.1:41667)
0929 11:25:40.610001 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:40.610015 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:40.610061 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:25:40.612418 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:40.612445 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:40.612943 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:40.613387 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:40.614638 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:40.616689 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:40.618357 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:40.619366 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:40.619544 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:40.623907 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:40.623915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:40.623928 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:40.623969 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:40.627031 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:40.627926 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:40.627931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:40.627936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:40.628083 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:40.628490 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:40.628493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:40.628494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:40.629094 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:40.629404 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:40.629848 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 11:25:40.630248 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":317,"threads_started":1}
I20260929 11:25:40.632918 21186 raft_consensus.cc:504] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60b37a21bee4e279ab70858f756e703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260929 11:25:40.633422 21186 leader_election.cc:302] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f60b37a21bee4e279ab70858f756e703; no voters:
I20260929 11:25:40.634127 21186 leader_election.cc:288] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:40.634234 21191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:40.635159 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:40.636545 21186 sys_catalog.cc:564] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:40.636941 21191 raft_consensus.cc:686] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [term 1 LEADER]: Becoming Leader. State: Replica: f60b37a21bee4e279ab70858f756e703, State: Running, Role: LEADER
I20260929 11:25:40.637552 21191 consensus_queue.cc:237] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60b37a21bee4e279ab70858f756e703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260929 11:25:40.642237 21193 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f60b37a21bee4e279ab70858f756e703. Latest consensus state: current_term: 1 leader_uuid: "f60b37a21bee4e279ab70858f756e703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60b37a21bee4e279ab70858f756e703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } } }
I20260929 11:25:40.642699 21193 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:40.645421 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f60b37a21bee4e279ab70858f756e703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60b37a21bee4e279ab70858f756e703" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } } }
I20260929 11:25:40.645800 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:40.653380 21202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:40.660655 21202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:40.662473 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:40.639732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50115 (local address 127.0.0.1:41667)
0929 11:25:40.639957 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:40.639967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:40.646266 (+ 6299us) server_negotiation.cc:407] Connection header received
0929 11:25:40.646491 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:40.646497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:40.646576 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:40.646676 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:40.648020 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:40.648927 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:40.650593 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:40.651036 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:40.651174 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:40.654170 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:40.654177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:40.654181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:40.654217 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:40.658362 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:40.659216 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:40.659220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:40.659223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:40.659366 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:40.659856 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:40.659860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:40.659863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:40.660262 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:40.661711 (+ 1449us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:40.661863 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 11:25:40.662093 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:25:40.670421 21202 catalog_manager.cc:1347] Generated new cluster ID: bd727c0c248d4c0d8159723b66156d77
I20260929 11:25:40.670552 21202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:40.696623 21202 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:40.697944 21202 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:40.710474 21202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f60b37a21bee4e279ab70858f756e703: Generated new TSK 0
I20260929 11:25:40.711329 21202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:40.721427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21119
I20260929 11:25:40.722502 21181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:40.814672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21119
2026-09-29T11:25:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:40 dist-test-slave-srst krb5kdc[21218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:40 dist-test-slave-srst krb5kdc[21218](info): set up 2 sockets
Sep 29 11:25:40 dist-test-slave-srst krb5kdc[21218](info): commencing operation
krb5kdc: starting...
W20260929 11:25:41.909256 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:42 dist-test-slave-srst krb5kdc[21218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:42.165781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:42.495707 21231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:42.496037 21231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:42.496207 21231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:42.503350 21231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:42.503468 21231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:42.503574 21231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:42.503674 21231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:42.503753 21231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:42.513715 21231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:42.514966 21231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:42.516351 21231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:42.528090 21237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:42.528038 21239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:42.528142 21236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:42.528910 21231 server_base.cc:1034] running on GCE node
I20260929 11:25:42.529703 21231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:42.531042 21231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:42.532238 21231 hybrid_clock.cc:648] HybridClock initialized: now 1790681142532181 us; error 53 us; skew 500 ppm
I20260929 11:25:42.532652 21231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:42 dist-test-slave-srst krb5kdc[21218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:42.535635 21231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:42.537147 21231 webserver.cc:460] Webserver started at http://127.0.0.1:43789/ using document root <none> and password file <none>
I20260929 11:25:42.537775 21231 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:42.537871 21231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:42.538120 21231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:42.540988 21231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "86bfab725ef9476fad614a026e4088d4"
format_stamp: "Formatted at 2026-09-29 11:25:42 on dist-test-slave-srst"
server_key: "93cb3691d7c9385c7d686b878551eeb2"
server_key_iv: "d0c0b0717de0df18dd7f31cae6c3e249"
server_key_version: "encryptionkey@0"
I20260929 11:25:42.541687 21231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "86bfab725ef9476fad614a026e4088d4"
format_stamp: "Formatted at 2026-09-29 11:25:42 on dist-test-slave-srst"
server_key: "93cb3691d7c9385c7d686b878551eeb2"
server_key_iv: "d0c0b0717de0df18dd7f31cae6c3e249"
server_key_version: "encryptionkey@0"
I20260929 11:25:42.546473 21231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:25:42.549770 21246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:42.551066 21231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:25:42.551263 21231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "86bfab725ef9476fad614a026e4088d4"
format_stamp: "Formatted at 2026-09-29 11:25:42 on dist-test-slave-srst"
server_key: "93cb3691d7c9385c7d686b878551eeb2"
server_key_iv: "d0c0b0717de0df18dd7f31cae6c3e249"
server_key_version: "encryptionkey@0"
I20260929 11:25:42.551465 21231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:42.571626 21231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:42.575559 21231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:42.575848 21231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:42.594385 21231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37627
I20260929 11:25:42.594403 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37627 every 8 connection(s)
I20260929 11:25:42.595783 21231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:42.599532 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21231
I20260929 11:25:42.599687 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:42.599735 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:42.600275 388 external_mini_cluster.cc:1428] Setting key b9e11cbbfde31276574241adaf7bc498
I20260929 11:25:42.612259 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4: Bootstrap starting.
I20260929 11:25:42.613070 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:42.602121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43907 (local address 127.0.0.1:37627)
0929 11:25:42.602911 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:42.602922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:42.602960 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:25:42.604580 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:42.604652 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:42.605047 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:42.605427 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:42.605973 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:42.607277 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:42.608708 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:42.609392 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:42.609594 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:42.610751 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:42.610759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:42.610770 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:42.611278 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:42.611354 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:42.611379 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:42.611501 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:42.611717 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 11:25:42.611970 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":352,"threads_started":1}
I20260929 11:25:42.615372 21298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:42.616571 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:42.616535 21298 log.cc:824] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:42.619119 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4: No bootstrap required, opened a new log
I20260929 11:25:42.626653 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:42.620681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54643 (local address 127.0.0.1:37627)
0929 11:25:42.620843 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:42.620846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:42.621305 (+ 459us) server_negotiation.cc:407] Connection header received
0929 11:25:42.621516 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:42.621519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:42.621578 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:42.621692 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:42.622187 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:42.622915 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:42.624202 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:42.624745 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:42.624887 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:42.625945 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:42.625950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:42.625953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:42.625976 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:42.626010 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:42.626024 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:42.626130 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:42.626289 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 11:25:42.626372 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:25:42.627255 21298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bfab725ef9476fad614a026e4088d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260929 11:25:42.627651 21298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:42.627779 21298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bfab725ef9476fad614a026e4088d4, State: Initialized, Role: FOLLOWER
I20260929 11:25:42.628412 21298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bfab725ef9476fad614a026e4088d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260929 11:25:42.628677 21298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:42.628824 21298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:42.629009 21298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:42.631345 21298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bfab725ef9476fad614a026e4088d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260929 11:25:42.631820 21298 leader_election.cc:302] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86bfab725ef9476fad614a026e4088d4; no voters:
I20260929 11:25:42.632357 21298 leader_election.cc:288] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:42.632498 21308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:42.634222 21308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [term 1 LEADER]: Becoming Leader. State: Replica: 86bfab725ef9476fad614a026e4088d4, State: Running, Role: LEADER
I20260929 11:25:42.634626 21298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:42.634785 21308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bfab725ef9476fad614a026e4088d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260929 11:25:42.637848 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bfab725ef9476fad614a026e4088d4. Latest consensus state: current_term: 1 leader_uuid: "86bfab725ef9476fad614a026e4088d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bfab725ef9476fad614a026e4088d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } } }
I20260929 11:25:42.637908 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86bfab725ef9476fad614a026e4088d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bfab725ef9476fad614a026e4088d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } } }
I20260929 11:25:42.638274 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:42.638273 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:42.641098 21318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:42.645925 21318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:42.654723 21318 catalog_manager.cc:1347] Generated new cluster ID: 6823ba8a691b49d6bb96349a654212a7
I20260929 11:25:42.654824 21318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:42.674273 21318 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:42.675660 21318 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:42.687016 21318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86bfab725ef9476fad614a026e4088d4: Generated new TSK 0
I20260929 11:25:42.687885 21318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:42.693167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21231
W20260929 11:25:42.694523 21293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43907
W20260929 11:25:42.695238 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:42.695396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21231
2026-09-29T11:25:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:42 dist-test-slave-srst krb5kdc[21329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:42 dist-test-slave-srst krb5kdc[21329](info): set up 2 sockets
Sep 29 11:25:42 dist-test-slave-srst krb5kdc[21329](info): commencing operation
krb5kdc: starting...
W20260929 11:25:43.785080 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:43 dist-test-slave-srst krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:44.029445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:44.364624 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:44.364897 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:44.365091 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:44.372232 21342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:44.372324 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:44.372368 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:44.372439 21342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:44.372501 21342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:44.381966 21342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:44.383291 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:44.384579 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:44.396605 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:44.396688 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:44.396660 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:44.397548 21342 server_base.cc:1034] running on GCE node
I20260929 11:25:44.398291 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:44.399617 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:44.400847 21342 hybrid_clock.cc:648] HybridClock initialized: now 1790681144400815 us; error 56 us; skew 500 ppm
I20260929 11:25:44.401202 21342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:44 dist-test-slave-srst krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:44.404182 21342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:44.405511 21342 webserver.cc:460] Webserver started at http://127.0.0.1:45587/ using document root <none> and password file <none>
I20260929 11:25:44.406128 21342 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:44.406239 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:44.406502 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:44.409332 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "d1da928a62684693b113ed7937e1b64d"
format_stamp: "Formatted at 2026-09-29 11:25:44 on dist-test-slave-srst"
server_key: "304e0325f84af7362bba5490f5a98748"
server_key_iv: "ad7df64908a82a0ce6225c713d55226b"
server_key_version: "encryptionkey@0"
I20260929 11:25:44.410041 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1da928a62684693b113ed7937e1b64d"
format_stamp: "Formatted at 2026-09-29 11:25:44 on dist-test-slave-srst"
server_key: "304e0325f84af7362bba5490f5a98748"
server_key_iv: "ad7df64908a82a0ce6225c713d55226b"
server_key_version: "encryptionkey@0"
I20260929 11:25:44.415042 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:25:44.418416 21357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:44.419709 21342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:25:44.419885 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "d1da928a62684693b113ed7937e1b64d"
format_stamp: "Formatted at 2026-09-29 11:25:44 on dist-test-slave-srst"
server_key: "304e0325f84af7362bba5490f5a98748"
server_key_iv: "ad7df64908a82a0ce6225c713d55226b"
server_key_version: "encryptionkey@0"
I20260929 11:25:44.420078 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:44.425806 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:44.429589 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:44.429889 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:44.448019 21342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36187
I20260929 11:25:44.448062 21408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36187 every 8 connection(s)
I20260929 11:25:44.449440 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:44.453265 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:44.453258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21342
I20260929 11:25:44.453444 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:44.453903 388 external_mini_cluster.cc:1428] Setting key 1a64290fd260dd1c01907ebadf83ad62
I20260929 11:25:44.462945 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:44.455729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41957 (local address 127.0.0.1:36187)
0929 11:25:44.456667 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:44.456682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:44.456733 (+ 51us) server_negotiation.cc:407] Connection header received
0929 11:25:44.459116 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:44.459145 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:44.459682 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:44.460225 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:44.461324 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:44.461330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:44.461342 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:44.461357 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:44.461407 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:44.461425 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:44.461696 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:44.461852 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 11:25:44.462053 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":351,"threads_started":1}
I20260929 11:25:44.466593 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:44.467931 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d: Bootstrap starting.
I20260929 11:25:44.471760 21409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:44.473125 21409 log.cc:824] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:44.473465 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:44.470961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39121 (local address 127.0.0.1:36187)
0929 11:25:44.471140 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:44.471148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:44.471484 (+ 336us) server_negotiation.cc:407] Connection header received
0929 11:25:44.471684 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:44.471691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:44.471742 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:44.471855 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:44.472846 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:44.472851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:44.472854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:44.472868 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:44.472906 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:44.472920 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:44.473040 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:44.473149 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 11:25:44.473206 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:25:44.476177 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d: No bootstrap required, opened a new log
I20260929 11:25:44.484095 21409 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da928a62684693b113ed7937e1b64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260929 11:25:44.484515 21409 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:44.484644 21409 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da928a62684693b113ed7937e1b64d, State: Initialized, Role: FOLLOWER
I20260929 11:25:44.485381 21409 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da928a62684693b113ed7937e1b64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260929 11:25:44.485702 21409 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:44.485868 21409 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:44.486056 21409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:44.488268 21409 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da928a62684693b113ed7937e1b64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260929 11:25:44.488859 21409 leader_election.cc:302] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1da928a62684693b113ed7937e1b64d; no voters:
I20260929 11:25:44.489442 21409 leader_election.cc:288] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:44.489547 21419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:44.490224 21419 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [term 1 LEADER]: Becoming Leader. State: Replica: d1da928a62684693b113ed7937e1b64d, State: Running, Role: LEADER
I20260929 11:25:44.490778 21419 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da928a62684693b113ed7937e1b64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } }
I20260929 11:25:44.491581 21409 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:44.493868 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1da928a62684693b113ed7937e1b64d. Latest consensus state: current_term: 1 leader_uuid: "d1da928a62684693b113ed7937e1b64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da928a62684693b113ed7937e1b64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } } }
I20260929 11:25:44.494457 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:44.495046 21420 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1da928a62684693b113ed7937e1b64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da928a62684693b113ed7937e1b64d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36187 } } }
I20260929 11:25:44.495464 21420 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:44.496711 21425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:44.501392 21425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:44.510171 21425 catalog_manager.cc:1347] Generated new cluster ID: ec85bd105a8e4e1b8caab05c3b564b1f
I20260929 11:25:44.510301 21425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:44.545814 21425 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:44.546314 21425 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:44.558172 21425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1da928a62684693b113ed7937e1b64d: Generated new TSK 0
I20260929 11:25:44.559267 21425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:44.579234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21342
W20260929 11:25:44.580366 21404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41957
W20260929 11:25:44.581097 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:44.581264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21342
2026-09-29T11:25:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:44 dist-test-slave-srst krb5kdc[21440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:44 dist-test-slave-srst krb5kdc[21440](info): set up 2 sockets
Sep 29 11:25:44 dist-test-slave-srst krb5kdc[21440](info): commencing operation
krb5kdc: starting...
W20260929 11:25:45.665114 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:45 dist-test-slave-srst krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:45.927215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:46.248483 21453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:46.248808 21453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:46.248915 21453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:46.255946 21453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:46.256047 21453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:46.256101 21453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:46.256155 21453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:46.256197 21453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:46.265888 21453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:46.267067 21453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:46.268352 21453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:46.279790 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:46.279838 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:46.279866 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:46.280786 21453 server_base.cc:1034] running on GCE node
I20260929 11:25:46.281474 21453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:46.282733 21453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:46.283918 21453 hybrid_clock.cc:648] HybridClock initialized: now 1790681146283864 us; error 45 us; skew 500 ppm
I20260929 11:25:46.284205 21453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:46 dist-test-slave-srst krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:46.287350 21453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:46.288875 21453 webserver.cc:460] Webserver started at http://127.0.0.1:38641/ using document root <none> and password file <none>
I20260929 11:25:46.289529 21453 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:46.289659 21453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:46.289983 21453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:46.292966 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "c268c5e05cac4433a1a95113058cd3e1"
format_stamp: "Formatted at 2026-09-29 11:25:46 on dist-test-slave-srst"
server_key: "dc59653c7c535ccdc3d8a1a29990ce4b"
server_key_iv: "99ffd7ae39b2effac9e4b9a378af93f2"
server_key_version: "encryptionkey@0"
I20260929 11:25:46.293700 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "c268c5e05cac4433a1a95113058cd3e1"
format_stamp: "Formatted at 2026-09-29 11:25:46 on dist-test-slave-srst"
server_key: "dc59653c7c535ccdc3d8a1a29990ce4b"
server_key_iv: "99ffd7ae39b2effac9e4b9a378af93f2"
server_key_version: "encryptionkey@0"
I20260929 11:25:46.298702 21453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:25:46.301994 21468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:46.303238 21453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:25:46.303408 21453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "c268c5e05cac4433a1a95113058cd3e1"
format_stamp: "Formatted at 2026-09-29 11:25:46 on dist-test-slave-srst"
server_key: "dc59653c7c535ccdc3d8a1a29990ce4b"
server_key_iv: "99ffd7ae39b2effac9e4b9a378af93f2"
server_key_version: "encryptionkey@0"
I20260929 11:25:46.303588 21453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:46.335146 21453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:46.339078 21453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:46.339356 21453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:46.356956 21453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33699
I20260929 11:25:46.356994 21519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33699 every 8 connection(s)
I20260929 11:25:46.358259 21453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:46.359906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21453
I20260929 11:25:46.360095 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:46.360507 388 external_mini_cluster.cc:1428] Setting key f6734f16567976e7e9f28b88b3bae461
I20260929 11:25:46.362305 21520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:25:46 dist-test-slave-srst krb5kdc[21440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:25:46.375566 21520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1: Bootstrap starting.
I20260929 11:25:46.379638 21520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:46.380865 21520 log.cc:824] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:46.384145 21520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1: No bootstrap required, opened a new log
I20260929 11:25:46.384570 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:46.362341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:50339 (local address 127.0.0.1:33699)
0929 11:25:46.363161 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:46.363175 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:46.363224 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:25:46.365327 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:46.365353 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:46.365888 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:46.366315 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:46.367484 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:46.369511 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:46.370777 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:46.371839 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:46.372036 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:46.376198 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:46.376208 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:46.376226 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:46.376268 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:46.380171 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:46.381026 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:46.381032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:46.381040 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:46.381248 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:46.381770 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:46.381773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:46.381775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:46.382319 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:46.382610 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:46.383013 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 11:25:46.383321 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":351,"threads_started":1}
I20260929 11:25:46.388453 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:46.392504 21520 raft_consensus.cc:348] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c268c5e05cac4433a1a95113058cd3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260929 11:25:46.393033 21520 raft_consensus.cc:374] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:46.393141 21520 raft_consensus.cc:729] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c268c5e05cac4433a1a95113058cd3e1, State: Initialized, Role: FOLLOWER
I20260929 11:25:46.393795 21520 consensus_queue.cc:260] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c268c5e05cac4433a1a95113058cd3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260929 11:25:46.394093 21520 raft_consensus.cc:388] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:46.394244 21520 raft_consensus.cc:482] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:46.394454 21520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:46.397401 21520 raft_consensus.cc:504] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c268c5e05cac4433a1a95113058cd3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260929 11:25:46.397926 21520 leader_election.cc:302] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c268c5e05cac4433a1a95113058cd3e1; no voters:
I20260929 11:25:46.398622 21520 leader_election.cc:288] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:46.398728 21530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:46.400961 21530 raft_consensus.cc:686] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [term 1 LEADER]: Becoming Leader. State: Replica: c268c5e05cac4433a1a95113058cd3e1, State: Running, Role: LEADER
I20260929 11:25:46.401077 21520 sys_catalog.cc:564] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:46.401660 21530 consensus_queue.cc:237] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c268c5e05cac4433a1a95113058cd3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } }
I20260929 11:25:46.404939 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c268c5e05cac4433a1a95113058cd3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c268c5e05cac4433a1a95113058cd3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } } }
I20260929 11:25:46.406205 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:46.405617 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c268c5e05cac4433a1a95113058cd3e1. Latest consensus state: current_term: 1 leader_uuid: "c268c5e05cac4433a1a95113058cd3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c268c5e05cac4433a1a95113058cd3e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33699 } } }
I20260929 11:25:46.406854 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:46.411092 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:46.392697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:36793 (local address 127.0.0.1:33699)
0929 11:25:46.392902 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:46.392910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:46.393282 (+ 372us) server_negotiation.cc:407] Connection header received
0929 11:25:46.393529 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:46.393536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:46.393614 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:46.393700 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:46.395619 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:46.396387 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:46.398187 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:46.398885 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:46.399042 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:46.401971 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:46.401975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:46.401979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:46.402010 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:46.405839 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:46.406968 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:46.406976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:46.406982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:46.407152 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:46.407869 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:46.407876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:46.407882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:46.408378 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:46.410050 (+ 1672us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:46.410294 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 11:25:46.410519 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:25:46.412714 21540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:46.417905 21540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:46.427614 21540 catalog_manager.cc:1347] Generated new cluster ID: e343d99aabd64a8481e4adcfc7a6d7e4
I20260929 11:25:46.427768 21540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:46.439798 21540 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:46.441030 21540 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:46.456291 21540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c268c5e05cac4433a1a95113058cd3e1: Generated new TSK 0
I20260929 11:25:46.457160 21540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:46.468884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21453
I20260929 11:25:46.469954 21515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:46.559738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21453
W20260929 11:25:46.579645 21438 connection.cc:535] client connection to 127.0.0.1:33699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33699): Connection reset by peer (error 104)
2026-09-29T11:25:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:46 dist-test-slave-srst krb5kdc[21552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:46 dist-test-slave-srst krb5kdc[21552](info): set up 2 sockets
Sep 29 11:25:46 dist-test-slave-srst krb5kdc[21552](info): commencing operation
krb5kdc: starting...
W20260929 11:25:47.653187 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:47 dist-test-slave-srst krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:47.899855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:48.227234 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:48.227511 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:48.227694 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:48.234911 21565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:48.235046 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:48.235172 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:48.235277 21565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:48.235342 21565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:48.244993 21565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:48.246320 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:48.247702 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:48.259428 21570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:48.259397 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:48.259481 21573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:48.260288 21565 server_base.cc:1034] running on GCE node
I20260929 11:25:48.261091 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:48.262524 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:48.263751 21565 hybrid_clock.cc:648] HybridClock initialized: now 1790681148263706 us; error 37 us; skew 500 ppm
I20260929 11:25:48.264081 21565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:48 dist-test-slave-srst krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:48.267138 21565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:48.268569 21565 webserver.cc:460] Webserver started at http://127.0.0.1:43435/ using document root <none> and password file <none>
I20260929 11:25:48.269275 21565 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:48.269413 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:48.269708 21565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:48.272650 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "d58e7bc8670a4dc4994a13c5938263a8"
format_stamp: "Formatted at 2026-09-29 11:25:48 on dist-test-slave-srst"
server_key: "131062af69a136bf50d7f621a573635b"
server_key_iv: "ce4b343b446495d7abb8295e80d1bdc2"
server_key_version: "encryptionkey@0"
I20260929 11:25:48.273415 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "d58e7bc8670a4dc4994a13c5938263a8"
format_stamp: "Formatted at 2026-09-29 11:25:48 on dist-test-slave-srst"
server_key: "131062af69a136bf50d7f621a573635b"
server_key_iv: "ce4b343b446495d7abb8295e80d1bdc2"
server_key_version: "encryptionkey@0"
I20260929 11:25:48.278273 21565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:25:48.281638 21580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:48.282917 21565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:25:48.283085 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "d58e7bc8670a4dc4994a13c5938263a8"
format_stamp: "Formatted at 2026-09-29 11:25:48 on dist-test-slave-srst"
server_key: "131062af69a136bf50d7f621a573635b"
server_key_iv: "ce4b343b446495d7abb8295e80d1bdc2"
server_key_version: "encryptionkey@0"
I20260929 11:25:48.283285 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:48.301997 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:48.305971 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:48.306273 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:48.324519 21565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41127
I20260929 11:25:48.324553 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41127 every 8 connection(s)
I20260929 11:25:48.325902 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:48.329530 21632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:48.333827 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21565
I20260929 11:25:48.334007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:48.334518 388 external_mini_cluster.cc:1428] Setting key 393a4885438b1c957afddc0b8f594971
I20260929 11:25:48.342231 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8: Bootstrap starting.
I20260929 11:25:48.345468 21632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:48.346635 21632 log.cc:824] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:48.347609 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:48.336460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:44727 (local address 127.0.0.1:41127)
0929 11:25:48.337238 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:48.337250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:48.337291 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:25:48.338877 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:48.338900 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:48.339329 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:48.339727 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:48.340287 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:48.341637 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:48.343027 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:48.343689 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:48.343870 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:48.345108 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:48.345115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:48.345129 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:48.345711 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:48.345786 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:48.345812 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:48.345975 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:48.346247 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:25:48.346419 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":321,"threads_started":1}
I20260929 11:25:48.349684 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8: No bootstrap required, opened a new log
I20260929 11:25:48.351415 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:48.357965 21632 raft_consensus.cc:348] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58e7bc8670a4dc4994a13c5938263a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260929 11:25:48.358429 21632 raft_consensus.cc:374] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:48.358544 21632 raft_consensus.cc:729] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58e7bc8670a4dc4994a13c5938263a8, State: Initialized, Role: FOLLOWER
I20260929 11:25:48.359182 21632 consensus_queue.cc:260] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58e7bc8670a4dc4994a13c5938263a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260929 11:25:48.359470 21632 raft_consensus.cc:388] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:48.359611 21632 raft_consensus.cc:482] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:48.359825 21632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:48.362285 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:48.355708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:41359 (local address 127.0.0.1:41127)
0929 11:25:48.355928 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:48.355933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:48.356229 (+ 296us) server_negotiation.cc:407] Connection header received
0929 11:25:48.356401 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:48.356408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:48.356467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:48.356547 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:48.357152 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:48.357847 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:48.359707 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:48.360229 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:48.360329 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:48.361542 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:48.361546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:48.361550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:48.361564 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:48.361618 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:48.361633 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:48.361718 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:48.361925 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 11:25:48.361963 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:25:48.362813 21632 raft_consensus.cc:504] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58e7bc8670a4dc4994a13c5938263a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260929 11:25:48.363298 21632 leader_election.cc:302] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d58e7bc8670a4dc4994a13c5938263a8; no voters:
I20260929 11:25:48.363974 21632 leader_election.cc:288] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:48.364058 21642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:48.364840 21642 raft_consensus.cc:686] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [term 1 LEADER]: Becoming Leader. State: Replica: d58e7bc8670a4dc4994a13c5938263a8, State: Running, Role: LEADER
I20260929 11:25:48.365432 21642 consensus_queue.cc:237] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58e7bc8670a4dc4994a13c5938263a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260929 11:25:48.366092 21632 sys_catalog.cc:564] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:48.367890 21643 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d58e7bc8670a4dc4994a13c5938263a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58e7bc8670a4dc4994a13c5938263a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } } }
I20260929 11:25:48.368290 21643 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:48.372879 21644 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58e7bc8670a4dc4994a13c5938263a8. Latest consensus state: current_term: 1 leader_uuid: "d58e7bc8670a4dc4994a13c5938263a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d58e7bc8670a4dc4994a13c5938263a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } } }
I20260929 11:25:48.373258 21644 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:48.373899 21648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:48.379102 21648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:48.388536 21648 catalog_manager.cc:1347] Generated new cluster ID: 0104cda3379d4f898bb43769050b6f47
I20260929 11:25:48.388715 21648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:48.422335 21648 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:48.423681 21648 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:48.433672 21648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d58e7bc8670a4dc4994a13c5938263a8: Generated new TSK 0
I20260929 11:25:48.434527 21648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:48.447609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21565
W20260929 11:25:48.449076 21627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:44727
W20260929 11:25:48.449877 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:48.450037 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21565
W20260929 11:25:48.466925 21550 connection.cc:535] client connection to 127.0.0.1:41127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41127): Connection reset by peer (error 104)
2026-09-29T11:25:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:48 dist-test-slave-srst krb5kdc[21663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:48 dist-test-slave-srst krb5kdc[21663](info): set up 2 sockets
Sep 29 11:25:48 dist-test-slave-srst krb5kdc[21663](info): commencing operation
krb5kdc: starting...
W20260929 11:25:49.557231 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:49 dist-test-slave-srst krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:49.809401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:50.136940 21676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:50.137209 21676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:50.137308 21676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:50.144292 21676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:50.144392 21676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:50.144441 21676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:50.144501 21676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:50.144541 21676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:50.154240 21676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:50.155436 21676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:50.156709 21676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:50.168036 21681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:50.168108 21682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:50.168073 21684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:50.168453 21676 server_base.cc:1034] running on GCE node
I20260929 11:25:50.169461 21676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:50.170804 21676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:50.172043 21676 hybrid_clock.cc:648] HybridClock initialized: now 1790681150171988 us; error 54 us; skew 500 ppm
I20260929 11:25:50.172374 21676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:50 dist-test-slave-srst krb5kdc[21663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:50.181475 21676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:50.182987 21676 webserver.cc:460] Webserver started at http://127.0.0.1:41339/ using document root <none> and password file <none>
I20260929 11:25:50.183678 21676 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:50.183823 21676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:50.184139 21676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:50.187062 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "f5d4706ae99c410da0ec4b10d646d8d7"
format_stamp: "Formatted at 2026-09-29 11:25:50 on dist-test-slave-srst"
server_key: "0bff077b35dc38d98dc72d4c58a7b8b2"
server_key_iv: "688dff99f32ba5af9bcad3de6ec01735"
server_key_version: "encryptionkey@0"
I20260929 11:25:50.187813 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "f5d4706ae99c410da0ec4b10d646d8d7"
format_stamp: "Formatted at 2026-09-29 11:25:50 on dist-test-slave-srst"
server_key: "0bff077b35dc38d98dc72d4c58a7b8b2"
server_key_iv: "688dff99f32ba5af9bcad3de6ec01735"
server_key_version: "encryptionkey@0"
I20260929 11:25:50.192756 21676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:25:50.195950 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:50.197250 21676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:50.197427 21676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "f5d4706ae99c410da0ec4b10d646d8d7"
format_stamp: "Formatted at 2026-09-29 11:25:50 on dist-test-slave-srst"
server_key: "0bff077b35dc38d98dc72d4c58a7b8b2"
server_key_iv: "688dff99f32ba5af9bcad3de6ec01735"
server_key_version: "encryptionkey@0"
I20260929 11:25:50.197621 21676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:50.203001 21676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:50.206756 21676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:50.207062 21676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:50.225373 21676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39185
I20260929 11:25:50.225420 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39185 every 8 connection(s)
I20260929 11:25:50.226688 21676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:50.230362 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:50.234109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21676
I20260929 11:25:50.234316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:50.234738 388 external_mini_cluster.cc:1428] Setting key 21d52d511ff612f3a7ed0766728d9298
I20260929 11:25:50.243108 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:50.236469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:54063 (local address 127.0.0.1:39185)
0929 11:25:50.237349 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:50.237363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:50.237411 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:25:50.239243 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:50.239275 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:50.239753 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:50.240206 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:50.241210 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:50.241233 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:50.241249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:50.241274 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:50.241350 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:50.241374 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:50.241592 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:50.241721 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 11:25:50.241908 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":318,"threads_started":1}
I20260929 11:25:50.243850 21743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7: Bootstrap starting.
I20260929 11:25:50.246635 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:50.247074 21743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:50.248281 21743 log.cc:824] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:50.251412 21743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7: No bootstrap required, opened a new log
I20260929 11:25:50.254063 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:50.251704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:49439 (local address 127.0.0.1:39185)
0929 11:25:50.251874 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:50.251881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:50.252222 (+ 341us) server_negotiation.cc:407] Connection header received
0929 11:25:50.252349 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:50.252354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:50.252404 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:50.252507 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:50.253415 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:50.253425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:50.253430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:50.253453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:50.253498 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:50.253524 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:50.253639 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:50.253714 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 11:25:50.253767 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:25:50.259358 21743 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39185 } }
I20260929 11:25:50.259703 21743 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:50.259861 21743 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d4706ae99c410da0ec4b10d646d8d7, State: Initialized, Role: FOLLOWER
I20260929 11:25:50.260505 21743 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39185 } }
I20260929 11:25:50.260848 21743 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:50.261013 21743 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:50.261189 21743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:50.263283 21743 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39185 } }
I20260929 11:25:50.263787 21743 leader_election.cc:302] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d4706ae99c410da0ec4b10d646d8d7; no voters:
I20260929 11:25:50.264367 21743 leader_election.cc:288] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:50.264497 21753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:50.265209 21753 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [term 1 LEADER]: Becoming Leader. State: Replica: f5d4706ae99c410da0ec4b10d646d8d7, State: Running, Role: LEADER
I20260929 11:25:50.265769 21753 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39185 } }
I20260929 11:25:50.266410 21743 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:50.268401 21754 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39185 } } }
I20260929 11:25:50.268857 21754 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:50.269011 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d4706ae99c410da0ec4b10d646d8d7. Latest consensus state: current_term: 1 leader_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d4706ae99c410da0ec4b10d646d8d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39185 } } }
I20260929 11:25:50.269685 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:50.270620 21760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:50.276162 21760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:50.284987 21760 catalog_manager.cc:1347] Generated new cluster ID: 6768dfd87d6a4ee7a110e4011f5403c2
I20260929 11:25:50.285101 21760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:50.301331 21760 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:50.301755 21760 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:50.311820 21760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d4706ae99c410da0ec4b10d646d8d7: Generated new TSK 0
I20260929 11:25:50.312570 21760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:50.321892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21676
W20260929 11:25:50.322996 21738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:54063
W20260929 11:25:50.323688 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:50.323822 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21676
2026-09-29T11:25:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:50 dist-test-slave-srst krb5kdc[21774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:50 dist-test-slave-srst krb5kdc[21774](info): set up 2 sockets
Sep 29 11:25:50 dist-test-slave-srst krb5kdc[21774](info): commencing operation
krb5kdc: starting...
W20260929 11:25:51.421164 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:51 dist-test-slave-srst krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:51.681769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:52.009361 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:52.009719 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:52.009902 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:52.017513 21787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:52.017678 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:52.017773 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:52.017881 21787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:52.017966 21787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:52.029073 21787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:52.030421 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:52.031766 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:52.043406 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:52.043484 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:52.043457 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:52.044657 21787 server_base.cc:1034] running on GCE node
I20260929 11:25:52.045441 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:52.046801 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:52.048031 21787 hybrid_clock.cc:648] HybridClock initialized: now 1790681152047990 us; error 52 us; skew 500 ppm
I20260929 11:25:52.048384 21787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:52 dist-test-slave-srst krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:52.051528 21787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:52.052989 21787 webserver.cc:460] Webserver started at http://127.0.0.1:40265/ using document root <none> and password file <none>
I20260929 11:25:52.053668 21787 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:52.053813 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:52.054127 21787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:52.057183 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "a81c2057882f4ce0959e1b36d734d198"
format_stamp: "Formatted at 2026-09-29 11:25:52 on dist-test-slave-srst"
server_key: "ddd5a71242dcd77c146728da471a63c8"
server_key_iv: "40ef0e1c2a23c8834290c77c70139cf9"
server_key_version: "encryptionkey@0"
I20260929 11:25:52.057969 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "a81c2057882f4ce0959e1b36d734d198"
format_stamp: "Formatted at 2026-09-29 11:25:52 on dist-test-slave-srst"
server_key: "ddd5a71242dcd77c146728da471a63c8"
server_key_iv: "40ef0e1c2a23c8834290c77c70139cf9"
server_key_version: "encryptionkey@0"
I20260929 11:25:52.063089 21787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 11:25:52.066636 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:52.067912 21787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:25:52.068116 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "a81c2057882f4ce0959e1b36d734d198"
format_stamp: "Formatted at 2026-09-29 11:25:52 on dist-test-slave-srst"
server_key: "ddd5a71242dcd77c146728da471a63c8"
server_key_iv: "40ef0e1c2a23c8834290c77c70139cf9"
server_key_version: "encryptionkey@0"
I20260929 11:25:52.068315 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:52.083382 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:52.087841 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:52.088140 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:52.106388 21787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43495
I20260929 11:25:52.106418 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43495 every 8 connection(s)
I20260929 11:25:52.107805 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:52.111591 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:52.115921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21787
I20260929 11:25:52.116140 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:52.116626 388 external_mini_cluster.cc:1428] Setting key f7ff8d3868f6fd563e4d02f06d3049e2
I20260929 11:25:52.124022 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198: Bootstrap starting.
I20260929 11:25:52.127705 21854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:52.129005 21854 log.cc:824] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:52.131935 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198: No bootstrap required, opened a new log
Sep 29 11:25:52 dist-test-slave-srst krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:25:52.142627 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:52.118512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:42661 (local address 127.0.0.1:43495)
0929 11:25:52.119276 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:52.119293 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:52.119343 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:25:52.122250 (+ 2907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:52.122280 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:52.122811 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:52.123258 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:52.124668 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:52.126678 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:52.128099 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:52.129197 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:52.129373 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:52.133572 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:52.133583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:52.133603 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:52.133656 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:52.137808 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:52.138764 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:52.138772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:52.138781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:52.138996 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:52.139402 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:52.139407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:52.139411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:52.140122 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:52.140476 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:52.140832 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 11:25:52.141208 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":319,"threads_started":1}
I20260929 11:25:52.143079 21854 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81c2057882f4ce0959e1b36d734d198" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260929 11:25:52.143467 21854 raft_consensus.cc:374] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:52.143572 21854 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81c2057882f4ce0959e1b36d734d198, State: Initialized, Role: FOLLOWER
I20260929 11:25:52.144250 21854 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81c2057882f4ce0959e1b36d734d198" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260929 11:25:52.144507 21854 raft_consensus.cc:388] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:52.144685 21854 raft_consensus.cc:482] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:52.144882 21854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:52.147014 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:52.147887 21854 raft_consensus.cc:504] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81c2057882f4ce0959e1b36d734d198" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260929 11:25:52.148417 21854 leader_election.cc:302] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a81c2057882f4ce0959e1b36d734d198; no voters:
I20260929 11:25:52.149046 21854 leader_election.cc:288] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:52.149199 21859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:52.151031 21854 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:52.158226 21859 raft_consensus.cc:686] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [term 1 LEADER]: Becoming Leader. State: Replica: a81c2057882f4ce0959e1b36d734d198, State: Running, Role: LEADER
I20260929 11:25:52.159040 21859 consensus_queue.cc:237] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81c2057882f4ce0959e1b36d734d198" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } }
I20260929 11:25:52.162663 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a81c2057882f4ce0959e1b36d734d198. Latest consensus state: current_term: 1 leader_uuid: "a81c2057882f4ce0959e1b36d734d198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81c2057882f4ce0959e1b36d734d198" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } } }
I20260929 11:25:52.163076 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:52.163592 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a81c2057882f4ce0959e1b36d734d198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81c2057882f4ce0959e1b36d734d198" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43495 } } }
I20260929 11:25:52.163888 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:52.166100 21877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:52.171978 21877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:52.177626 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:52.160622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:42867 (local address 127.0.0.1:43495)
0929 11:25:52.160871 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:52.160881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:52.161624 (+ 743us) server_negotiation.cc:407] Connection header received
0929 11:25:52.162003 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:52.162015 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:52.162127 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:52.162216 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:52.164952 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:52.166475 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:52.168248 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:52.168816 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:52.168968 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:52.171487 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:52.171494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:52.171504 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:52.171536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:52.174752 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:52.175560 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:52.175567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:52.175576 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:52.175692 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:52.176152 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:52.176162 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:52.176164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:52.176465 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:52.176765 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:52.177123 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 11:25:52.177305 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:25:52.181910 21877 catalog_manager.cc:1347] Generated new cluster ID: b90458a7613e4ff08773f64562d6c19d
I20260929 11:25:52.182010 21877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:52.196776 21877 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:52.198596 21877 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:52.219657 21877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a81c2057882f4ce0959e1b36d734d198: Generated new TSK 0
I20260929 11:25:52.220829 21877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:52.229431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21787
I20260929 11:25:52.230927 21849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:52.320457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21787
W20260929 11:25:52.335260 21772 connection.cc:535] client connection to 127.0.0.1:43495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43495): Connection reset by peer (error 104)
2026-09-29T11:25:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:52 dist-test-slave-srst krb5kdc[21887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:52 dist-test-slave-srst krb5kdc[21887](info): set up 2 sockets
Sep 29 11:25:52 dist-test-slave-srst krb5kdc[21887](info): commencing operation
krb5kdc: starting...
W20260929 11:25:53.433244 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:53 dist-test-slave-srst krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:53.679646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:54.011219 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:54.011593 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:54.011780 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:54.019032 21900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:54.019183 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:54.019290 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:54.019400 21900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:54.019485 21900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:54.029476 21900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:54.030797 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:54.032157 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:54.043678 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:54.043771 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:54.043859 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:54.045099 21900 server_base.cc:1034] running on GCE node
I20260929 11:25:54.045958 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:54.047338 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:54.048569 21900 hybrid_clock.cc:648] HybridClock initialized: now 1790681154048528 us; error 57 us; skew 500 ppm
I20260929 11:25:54.048970 21900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:54 dist-test-slave-srst krb5kdc[21887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:54.051964 21900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:54.053445 21900 webserver.cc:460] Webserver started at http://127.0.0.1:35077/ using document root <none> and password file <none>
I20260929 11:25:54.054200 21900 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:54.054343 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:54.054644 21900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:54.057572 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "8681e84b42404846b84b86ec94f703e0"
format_stamp: "Formatted at 2026-09-29 11:25:54 on dist-test-slave-srst"
server_key: "3da77d146d113d031cd00094524bf0ed"
server_key_iv: "419c00f3c9334529f47b89cf9d771bf8"
server_key_version: "encryptionkey@0"
I20260929 11:25:54.058328 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "8681e84b42404846b84b86ec94f703e0"
format_stamp: "Formatted at 2026-09-29 11:25:54 on dist-test-slave-srst"
server_key: "3da77d146d113d031cd00094524bf0ed"
server_key_iv: "419c00f3c9334529f47b89cf9d771bf8"
server_key_version: "encryptionkey@0"
I20260929 11:25:54.063560 21900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:25:54.067215 21915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:54.068759 21900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:25:54.068954 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "8681e84b42404846b84b86ec94f703e0"
format_stamp: "Formatted at 2026-09-29 11:25:54 on dist-test-slave-srst"
server_key: "3da77d146d113d031cd00094524bf0ed"
server_key_iv: "419c00f3c9334529f47b89cf9d771bf8"
server_key_version: "encryptionkey@0"
I20260929 11:25:54.069149 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:54.085490 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:54.089529 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:54.089828 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:54.108083 21900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45065
I20260929 11:25:54.108120 21966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45065 every 8 connection(s)
I20260929 11:25:54.109467 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:54.113279 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:54.113332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 21900
I20260929 11:25:54.113511 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:54.113937 388 external_mini_cluster.cc:1428] Setting key 178d573e473b172936fa2abe7861dac7
I20260929 11:25:54.128427 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:54.115700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:47821 (local address 127.0.0.1:45065)
0929 11:25:54.116564 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:54.116581 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:54.116662 (+ 81us) server_negotiation.cc:407] Connection header received
0929 11:25:54.118390 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:54.118420 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:54.118987 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:54.119431 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:54.119971 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:54.121972 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:54.123289 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:54.124307 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:54.124491 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:54.125725 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:54.125737 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:54.125754 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:54.126493 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:54.126584 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:54.126615 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:54.126792 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:54.126981 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 11:25:54.127140 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":328,"threads_started":1}
I20260929 11:25:54.128973 21967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0: Bootstrap starting.
I20260929 11:25:54.132159 21967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:54.132232 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:54.133450 21967 log.cc:824] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:54.136689 21967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0: No bootstrap required, opened a new log
I20260929 11:25:54.144160 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:54.137220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:59043 (local address 127.0.0.1:45065)
0929 11:25:54.137424 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:54.137432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:54.137928 (+ 496us) server_negotiation.cc:407] Connection header received
0929 11:25:54.138121 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:54.138125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:54.138185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:54.138268 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:54.138898 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:54.139683 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:54.141634 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:54.142092 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:54.142191 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:54.143381 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:54.143385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:54.143389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:54.143404 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:54.143440 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:54.143455 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:54.143537 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:54.143748 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 11:25:54.143813 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:25:54.146624 21967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8681e84b42404846b84b86ec94f703e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260929 11:25:54.146999 21967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:54.147120 21967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8681e84b42404846b84b86ec94f703e0, State: Initialized, Role: FOLLOWER
I20260929 11:25:54.147697 21967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8681e84b42404846b84b86ec94f703e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260929 11:25:54.147940 21967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:54.148093 21967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:54.148257 21967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:54.150807 21967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8681e84b42404846b84b86ec94f703e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260929 11:25:54.151300 21967 leader_election.cc:302] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8681e84b42404846b84b86ec94f703e0; no voters:
I20260929 11:25:54.151858 21967 leader_election.cc:288] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:54.152017 21977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:54.153319 21977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8681e84b42404846b84b86ec94f703e0, State: Running, Role: LEADER
I20260929 11:25:54.153843 21977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8681e84b42404846b84b86ec94f703e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260929 11:25:54.154361 21967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:54.156734 21978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8681e84b42404846b84b86ec94f703e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8681e84b42404846b84b86ec94f703e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } } }
I20260929 11:25:54.156827 21979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8681e84b42404846b84b86ec94f703e0. Latest consensus state: current_term: 1 leader_uuid: "8681e84b42404846b84b86ec94f703e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8681e84b42404846b84b86ec94f703e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } } }
I20260929 11:25:54.157065 21978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:54.157217 21979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:54.159544 21985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:54.164477 21985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:54.173851 21985 catalog_manager.cc:1347] Generated new cluster ID: 0076e15c01c64269b05562079899ec9a
I20260929 11:25:54.173956 21985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:54.201328 21985 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:54.203130 21985 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:54.212916 21985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8681e84b42404846b84b86ec94f703e0: Generated new TSK 0
I20260929 11:25:54.213814 21985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:54.232271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21900
W20260929 11:25:54.233400 21962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:47821
W20260929 11:25:54.234130 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 21900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:54.234274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 21900
W20260929 11:25:54.248428 21885 connection.cc:535] client connection to 127.0.0.1:45065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45065): Connection reset by peer (error 104)
2026-09-29T11:25:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:54 dist-test-slave-srst krb5kdc[21998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:54 dist-test-slave-srst krb5kdc[21998](info): set up 2 sockets
Sep 29 11:25:54 dist-test-slave-srst krb5kdc[21998](info): commencing operation
krb5kdc: starting...
W20260929 11:25:55.333031 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:55 dist-test-slave-srst krb5kdc[21998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:25:55.583279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:55.904038 22011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:55.904309 22011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:55.904398 22011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:55.911247 22011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:55.911347 22011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:55.911391 22011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:55.911448 22011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:55.911487 22011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:55.920857 22011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:55.922020 22011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:55.923242 22011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:55.935045 22016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:55.934983 22017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:55.935011 22019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:55.935822 22011 server_base.cc:1034] running on GCE node
I20260929 11:25:55.936691 22011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:55.938068 22011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:55.939301 22011 hybrid_clock.cc:648] HybridClock initialized: now 1790681155939259 us; error 55 us; skew 500 ppm
I20260929 11:25:55.939653 22011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:55 dist-test-slave-srst krb5kdc[21998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:55.946700 22011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:25:55.948153 22011 webserver.cc:460] Webserver started at http://127.0.0.1:40139/ using document root <none> and password file <none>
I20260929 11:25:55.948922 22011 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:55.949066 22011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:55.949359 22011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:55.952378 22011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "bc984be267b4440ab1ff38f7fc296ee1"
format_stamp: "Formatted at 2026-09-29 11:25:55 on dist-test-slave-srst"
server_key: "2d0e964755d6d87febfd9727c7be507a"
server_key_iv: "11ad8ae59963aecbcd39902fc523e453"
server_key_version: "encryptionkey@0"
I20260929 11:25:55.953181 22011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc984be267b4440ab1ff38f7fc296ee1"
format_stamp: "Formatted at 2026-09-29 11:25:55 on dist-test-slave-srst"
server_key: "2d0e964755d6d87febfd9727c7be507a"
server_key_iv: "11ad8ae59963aecbcd39902fc523e453"
server_key_version: "encryptionkey@0"
I20260929 11:25:55.958006 22011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 11:25:55.961395 22026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:55.962699 22011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:25:55.962881 22011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "bc984be267b4440ab1ff38f7fc296ee1"
format_stamp: "Formatted at 2026-09-29 11:25:55 on dist-test-slave-srst"
server_key: "2d0e964755d6d87febfd9727c7be507a"
server_key_iv: "11ad8ae59963aecbcd39902fc523e453"
server_key_version: "encryptionkey@0"
I20260929 11:25:55.963073 22011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:55.968461 22011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:55.972147 22011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:55.972414 22011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:55.990267 22011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45071
I20260929 11:25:55.990345 22077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45071 every 8 connection(s)
I20260929 11:25:55.991564 22011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:55.995123 22078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:55.996353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22011
I20260929 11:25:55.996542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:55.997150 388 external_mini_cluster.cc:1428] Setting key 0724bc6d7ffcf255c1d7bd0ded947a50
I20260929 11:25:56.009462 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:55.999426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:51041 (local address 127.0.0.1:45071)
0929 11:25:56.002496 (+ 3070us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:56.002510 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:56.002556 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:25:56.005069 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:56.005100 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:56.005569 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:56.006002 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:56.007294 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:56.007304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:56.007329 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:56.007353 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:56.007444 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:56.007475 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:56.007756 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:56.007902 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 11:25:56.008121 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2782,"thread_start_us":2509,"threads_started":1}
I20260929 11:25:56.011487 22078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1: Bootstrap starting.
I20260929 11:25:56.013116 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:56.014981 22078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:56.016175 22078 log.cc:824] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:56.019600 22078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1: No bootstrap required, opened a new log
I20260929 11:25:56.020284 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:56.017728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:49663 (local address 127.0.0.1:45071)
0929 11:25:56.017989 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:56.017998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:56.018310 (+ 312us) server_negotiation.cc:407] Connection header received
0929 11:25:56.018471 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:56.018475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:56.018521 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:56.018612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:56.019551 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:56.019555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:56.019558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:56.019572 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:56.019610 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:56.019623 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:56.019771 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:56.019929 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 11:25:56.019977 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:25:56.026991 22078 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc984be267b4440ab1ff38f7fc296ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 11:25:56.027352 22078 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:56.027451 22078 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc984be267b4440ab1ff38f7fc296ee1, State: Initialized, Role: FOLLOWER
I20260929 11:25:56.028026 22078 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc984be267b4440ab1ff38f7fc296ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 11:25:56.028256 22078 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:56.028390 22078 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:56.028564 22078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:56.030835 22078 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc984be267b4440ab1ff38f7fc296ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 11:25:56.031330 22078 leader_election.cc:302] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc984be267b4440ab1ff38f7fc296ee1; no voters:
I20260929 11:25:56.031909 22078 leader_election.cc:288] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:56.032063 22088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:56.032779 22088 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [term 1 LEADER]: Becoming Leader. State: Replica: bc984be267b4440ab1ff38f7fc296ee1, State: Running, Role: LEADER
I20260929 11:25:56.033327 22088 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc984be267b4440ab1ff38f7fc296ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260929 11:25:56.033985 22078 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:56.036098 22090 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc984be267b4440ab1ff38f7fc296ee1. Latest consensus state: current_term: 1 leader_uuid: "bc984be267b4440ab1ff38f7fc296ee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc984be267b4440ab1ff38f7fc296ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } } }
I20260929 11:25:56.036533 22090 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:56.037096 22089 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc984be267b4440ab1ff38f7fc296ee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc984be267b4440ab1ff38f7fc296ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } } }
I20260929 11:25:56.037380 22089 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:56.038123 22095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:56.044117 22095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:56.052925 22095 catalog_manager.cc:1347] Generated new cluster ID: 4c8f62c6f79642d1a20574b2731aed1a
I20260929 11:25:56.053063 22095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:56.077582 22095 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:56.078001 22095 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:56.086122 22095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc984be267b4440ab1ff38f7fc296ee1: Generated new TSK 0
I20260929 11:25:56.086879 22095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:56.099268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22011
W20260929 11:25:56.100267 22073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:51041
W20260929 11:25:56.100989 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22011: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:56.101128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22011
2026-09-29T11:25:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:56 dist-test-slave-srst krb5kdc[22109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:56 dist-test-slave-srst krb5kdc[22109](info): set up 2 sockets
Sep 29 11:25:56 dist-test-slave-srst krb5kdc[22109](info): commencing operation
krb5kdc: starting...
W20260929 11:25:57.189091 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:57 dist-test-slave-srst krb5kdc[22109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:25:57.441941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35699
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:57.773959 22122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:57.774245 22122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:57.774336 22122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:57.781517 22122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:57.781628 22122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:57.781672 22122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:57.781733 22122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:57.781775 22122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:57.791491 22122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:57.792752 22122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:57.794039 22122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:57.805975 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:57.806052 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:57.806028 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:57.806802 22122 server_base.cc:1034] running on GCE node
I20260929 11:25:57.807580 22122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:57.809029 22122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:57.810263 22122 hybrid_clock.cc:648] HybridClock initialized: now 1790681157810213 us; error 28 us; skew 500 ppm
I20260929 11:25:57.810636 22122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:57 dist-test-slave-srst krb5kdc[22109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:57.813882 22122 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:25:57.815820 22122 webserver.cc:460] Webserver started at http://127.0.97.62:41133/ using document root <none> and password file <none>
I20260929 11:25:57.816521 22122 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:57.816681 22122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:57.817024 22122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:57.820081 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "0f75bf60ede34b9694497c83b245b142"
format_stamp: "Formatted at 2026-09-29 11:25:57 on dist-test-slave-srst"
server_key: "312fd985d86dac1d64c4c1a34e3d1007"
server_key_iv: "5059c22f5e32ae80733f973f359e1986"
server_key_version: "encryptionkey@0"
I20260929 11:25:57.820960 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f75bf60ede34b9694497c83b245b142"
format_stamp: "Formatted at 2026-09-29 11:25:57 on dist-test-slave-srst"
server_key: "312fd985d86dac1d64c4c1a34e3d1007"
server_key_iv: "5059c22f5e32ae80733f973f359e1986"
server_key_version: "encryptionkey@0"
I20260929 11:25:57.826315 22122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:25:57.829945 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:57.831269 22122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:25:57.831476 22122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "0f75bf60ede34b9694497c83b245b142"
format_stamp: "Formatted at 2026-09-29 11:25:57 on dist-test-slave-srst"
server_key: "312fd985d86dac1d64c4c1a34e3d1007"
server_key_iv: "5059c22f5e32ae80733f973f359e1986"
server_key_version: "encryptionkey@0"
I20260929 11:25:57.831692 22122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:57.850764 22122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:57.855366 22122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:57.855660 22122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:57.874193 22122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35699
I20260929 11:25:57.874217 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35699 every 8 connection(s)
I20260929 11:25:57.875566 22122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:57.876566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22122
I20260929 11:25:57.876799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:57.877303 388 external_mini_cluster.cc:1428] Setting key 1b05f3aff24786374eeeeb8964173a2d
I20260929 11:25:57.880069 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:25:57 dist-test-slave-srst krb5kdc[22109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:25:57.890360 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142: Bootstrap starting.
I20260929 11:25:57.893450 22189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:57.894513 22189 log.cc:824] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:57.897841 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142: No bootstrap required, opened a new log
I20260929 11:25:57.899454 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:57.878999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33655 (local address 127.0.97.62:35699)
0929 11:25:57.879727 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:57.879738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:57.879780 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:25:57.881514 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:57.881536 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:57.881911 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:57.882293 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:57.883755 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:57.885184 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:57.886450 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:57.887139 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:57.887358 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:57.891428 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:57.891435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:57.891449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:57.891489 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:57.894609 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:57.895668 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:57.895674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:57.895680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:57.895897 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:57.896344 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:57.896349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:57.896352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:57.897027 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:57.897340 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:57.897736 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 11:25:57.898072 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260929 11:25:57.903375 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:57.906417 22189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f75bf60ede34b9694497c83b245b142" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35699 } }
I20260929 11:25:57.906797 22189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:57.906919 22189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f75bf60ede34b9694497c83b245b142, State: Initialized, Role: FOLLOWER
I20260929 11:25:57.907490 22189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f75bf60ede34b9694497c83b245b142" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35699 } }
I20260929 11:25:57.907780 22189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:57.907922 22189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:57.908123 22189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:57.910799 22189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f75bf60ede34b9694497c83b245b142" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35699 } }
I20260929 11:25:57.911340 22189 leader_election.cc:302] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f75bf60ede34b9694497c83b245b142; no voters:
I20260929 11:25:57.911924 22189 leader_election.cc:288] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:57.912065 22199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:57.912734 22199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [term 1 LEADER]: Becoming Leader. State: Replica: 0f75bf60ede34b9694497c83b245b142, State: Running, Role: LEADER
I20260929 11:25:57.913487 22199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f75bf60ede34b9694497c83b245b142" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35699 } }
I20260929 11:25:57.915016 22189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:57.916357 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f75bf60ede34b9694497c83b245b142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f75bf60ede34b9694497c83b245b142" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35699 } } }
I20260929 11:25:57.916743 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:57.917408 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f75bf60ede34b9694497c83b245b142. Latest consensus state: current_term: 1 leader_uuid: "0f75bf60ede34b9694497c83b245b142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f75bf60ede34b9694497c83b245b142" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35699 } } }
I20260929 11:25:57.917877 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:57.924551 22204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:57.925715 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:57.907715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56149 (local address 127.0.97.62:35699)
0929 11:25:57.907899 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:57.907907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:57.908453 (+ 546us) server_negotiation.cc:407] Connection header received
0929 11:25:57.908565 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:57.908573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:57.908670 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:57.908785 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:57.910110 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:57.911251 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:57.913144 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:57.913845 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:57.913967 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:57.917136 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:57.917146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:57.917153 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:25:57.917219 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:57.921990 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:57.923079 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:57.923086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:57.923091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:57.923389 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:25:57.924049 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:25:57.924054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:25:57.924059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:25:57.924470 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:57.924888 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:57.925179 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:25:57.925356 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:25:57.929781 22204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:57.938868 22204 catalog_manager.cc:1347] Generated new cluster ID: 5432c4698b9f4aa5af523f49c6e6d075
I20260929 11:25:57.939002 22204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:57.955710 22204 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:57.957129 22204 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:57.965667 22204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f75bf60ede34b9694497c83b245b142: Generated new TSK 0
I20260929 11:25:57.966502 22204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:57.976912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22122
I20260929 11:25:57.978551 22184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:25:58.072650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22122
W20260929 11:25:58.090160 22107 connection.cc:535] client connection to 127.0.97.62:35699 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35699): Connection reset by peer (error 104)
2026-09-29T11:25:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:25:58 dist-test-slave-srst krb5kdc[22221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:25:58 dist-test-slave-srst krb5kdc[22221](info): set up 2 sockets
Sep 29 11:25:58 dist-test-slave-srst krb5kdc[22221](info): commencing operation
krb5kdc: starting...
W20260929 11:25:59.169132 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:25:59 dist-test-slave-srst krb5kdc[22221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:25:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:25:59.415834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37553
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:25:59.745211 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:25:59.745528 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:25:59.745748 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:25:59.753130 22234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:25:59.753278 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:25:59.753413 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:25:59.753530 22234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:25:59.753628 22234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:25:59.763849 22234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:25:59.765281 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:25:59.766742 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:25:59.779556 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:59.779626 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:25:59.779733 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:25:59.780642 22234 server_base.cc:1034] running on GCE node
I20260929 11:25:59.781503 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:25:59.782984 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:25:59.784217 22234 hybrid_clock.cc:648] HybridClock initialized: now 1790681159784164 us; error 42 us; skew 500 ppm
I20260929 11:25:59.784564 22234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:25:59 dist-test-slave-srst krb5kdc[22221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:25:59.787750 22234 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:25:59.789135 22234 webserver.cc:460] Webserver started at http://127.0.97.62:41845/ using document root <none> and password file <none>
I20260929 11:25:59.789810 22234 fs_manager.cc:359] Metadata directory not provided
I20260929 11:25:59.789942 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:25:59.790200 22234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:25:59.793157 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "be0469dca89f4fbfad0673b2f296e8e1"
format_stamp: "Formatted at 2026-09-29 11:25:59 on dist-test-slave-srst"
server_key: "9cf2df1d1c2eb06ed462e29447da052a"
server_key_iv: "fc70f4737c59a5a86265ea3ab4a9b06f"
server_key_version: "encryptionkey@0"
I20260929 11:25:59.793870 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "be0469dca89f4fbfad0673b2f296e8e1"
format_stamp: "Formatted at 2026-09-29 11:25:59 on dist-test-slave-srst"
server_key: "9cf2df1d1c2eb06ed462e29447da052a"
server_key_iv: "fc70f4737c59a5a86265ea3ab4a9b06f"
server_key_version: "encryptionkey@0"
I20260929 11:25:59.798813 22234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 11:25:59.802217 22249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:25:59.803515 22234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 11:25:59.803702 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "be0469dca89f4fbfad0673b2f296e8e1"
format_stamp: "Formatted at 2026-09-29 11:25:59 on dist-test-slave-srst"
server_key: "9cf2df1d1c2eb06ed462e29447da052a"
server_key_iv: "fc70f4737c59a5a86265ea3ab4a9b06f"
server_key_version: "encryptionkey@0"
I20260929 11:25:59.803892 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:25:59.853013 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:25:59.857952 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:25:59.858265 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:25:59.876767 22234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37553
I20260929 11:25:59.876787 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37553 every 8 connection(s)
I20260929 11:25:59.878155 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:25:59.881662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22234
I20260929 11:25:59.881788 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:25:59.881877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:25:59.882334 388 external_mini_cluster.cc:1428] Setting key b6d8f53736049a44fe48c8be6df02f00
I20260929 11:25:59.895601 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:59.883836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59277 (local address 127.0.97.62:37553)
0929 11:25:59.884632 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:59.884648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:59.884696 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:25:59.886887 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:59.886907 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:59.887303 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:59.887626 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:59.888205 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:59.889559 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:59.891345 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:59.891942 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:59.892092 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:59.893419 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:59.893429 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:59.893442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:59.893986 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:59.894061 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:59.894087 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:59.894207 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:59.894490 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 11:25:59.894665 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260929 11:25:59.896054 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1: Bootstrap starting.
I20260929 11:25:59.899405 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:25:59.899592 22301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1: Neither blocks nor log segments found. Creating new log.
I20260929 11:25:59.900561 22301 log.cc:824] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:25:59.903424 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1: No bootstrap required, opened a new log
I20260929 11:25:59.910301 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:25:59.903990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58425 (local address 127.0.97.62:37553)
0929 11:25:59.904176 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:25:59.904184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:25:59.904618 (+ 434us) server_negotiation.cc:407] Connection header received
0929 11:25:59.904831 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:25:59.904834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:25:59.904894 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:25:59.904994 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:25:59.905494 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:59.906291 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:59.907704 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:25:59.908207 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:25:59.908350 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:25:59.909589 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:25:59.909593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:25:59.909597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:25:59.909616 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:25:59.909654 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:25:59.909671 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:25:59.909804 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:25:59.909996 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 11:25:59.910052 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 11:25:59.912329 22301 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0469dca89f4fbfad0673b2f296e8e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260929 11:25:59.912701 22301 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:25:59.912820 22301 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0469dca89f4fbfad0673b2f296e8e1, State: Initialized, Role: FOLLOWER
I20260929 11:25:59.913445 22301 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0469dca89f4fbfad0673b2f296e8e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260929 11:25:59.913686 22301 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:25:59.913828 22301 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:25:59.914011 22301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:25:59.916064 22301 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0469dca89f4fbfad0673b2f296e8e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260929 11:25:59.916492 22301 leader_election.cc:302] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0469dca89f4fbfad0673b2f296e8e1; no voters:
I20260929 11:25:59.917223 22301 leader_election.cc:288] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:25:59.917411 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:25:59.918727 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [term 1 LEADER]: Becoming Leader. State: Replica: be0469dca89f4fbfad0673b2f296e8e1, State: Running, Role: LEADER
I20260929 11:25:59.919266 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0469dca89f4fbfad0673b2f296e8e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } }
I20260929 11:25:59.919454 22301 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:25:59.921774 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be0469dca89f4fbfad0673b2f296e8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0469dca89f4fbfad0673b2f296e8e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } } }
I20260929 11:25:59.922135 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:59.922616 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0469dca89f4fbfad0673b2f296e8e1. Latest consensus state: current_term: 1 leader_uuid: "be0469dca89f4fbfad0673b2f296e8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0469dca89f4fbfad0673b2f296e8e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37553 } } }
I20260929 11:25:59.922925 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:25:59.924973 22317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:25:59.931319 22317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:25:59.939183 22317 catalog_manager.cc:1347] Generated new cluster ID: 260399db29f8450fa6fc023c2c1f5a94
I20260929 11:25:59.939285 22317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:25:59.947896 22317 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:25:59.949733 22317 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:25:59.966646 22317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0469dca89f4fbfad0673b2f296e8e1: Generated new TSK 0
I20260929 11:25:59.967711 22317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:25:59.972955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22234
W20260929 11:25:59.974051 22296 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59277
W20260929 11:25:59.974812 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22234: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:25:59.974965 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22234
W20260929 11:25:59.991900 22219 connection.cc:535] client connection to 127.0.97.62:37553 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37553): Connection reset by peer (error 104)
2026-09-29T11:25:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:00 dist-test-slave-srst krb5kdc[22333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:00 dist-test-slave-srst krb5kdc[22333](info): set up 2 sockets
Sep 29 11:26:00 dist-test-slave-srst krb5kdc[22333](info): commencing operation
krb5kdc: starting...
W20260929 11:26:01.101030 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:26:01 dist-test-slave-srst krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:26:01.352011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33415
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:01.685228 22346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:01.685626 22346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:01.685817 22346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:01.693192 22346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:01.693352 22346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:01.693472 22346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:01.693612 22346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:01.693715 22346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:01.704263 22346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:01.705639 22346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:01.707015 22346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:01.719156 22354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:01.719275 22351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:01.719108 22352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:01.720247 22346 server_base.cc:1034] running on GCE node
I20260929 11:26:01.721094 22346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:01.722517 22346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:01.723781 22346 hybrid_clock.cc:648] HybridClock initialized: now 1790681161723736 us; error 55 us; skew 500 ppm
I20260929 11:26:01.724150 22346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:01 dist-test-slave-srst krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:01.733745 22346 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:26:01.735206 22346 webserver.cc:460] Webserver started at http://127.0.97.62:37451/ using document root <none> and password file <none>
I20260929 11:26:01.735828 22346 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:01.735925 22346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:01.736181 22346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:01.739104 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "e16a942044c24986abce44ced1567500"
format_stamp: "Formatted at 2026-09-29 11:26:01 on dist-test-slave-srst"
server_key: "f25acbd26344627f65e9d2075f9986ca"
server_key_iv: "a26a5b9d05c292f3269c708d26b58365"
server_key_version: "encryptionkey@0"
I20260929 11:26:01.739782 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "e16a942044c24986abce44ced1567500"
format_stamp: "Formatted at 2026-09-29 11:26:01 on dist-test-slave-srst"
server_key: "f25acbd26344627f65e9d2075f9986ca"
server_key_iv: "a26a5b9d05c292f3269c708d26b58365"
server_key_version: "encryptionkey@0"
I20260929 11:26:01.744829 22346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:26:01.748067 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:01.749393 22346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:01.749565 22346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "e16a942044c24986abce44ced1567500"
format_stamp: "Formatted at 2026-09-29 11:26:01 on dist-test-slave-srst"
server_key: "f25acbd26344627f65e9d2075f9986ca"
server_key_iv: "a26a5b9d05c292f3269c708d26b58365"
server_key_version: "encryptionkey@0"
I20260929 11:26:01.749758 22346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:01.755306 22346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:01.759188 22346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:01.759496 22346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:01.777328 22346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33415
I20260929 11:26:01.777369 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33415 every 8 connection(s)
I20260929 11:26:01.778688 22346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:01.782339 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:01.785261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22346
I20260929 11:26:01.785455 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:26:01.786019 388 external_mini_cluster.cc:1428] Setting key d870e1f8496e48554fc3f82d75b3ace0
I20260929 11:26:01.793651 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:01.787870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35951 (local address 127.0.97.62:33415)
0929 11:26:01.788681 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:01.788693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:01.788733 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:26:01.790347 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:01.790369 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:01.790697 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:01.791084 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:01.792096 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:01.792102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:01.792115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:01.792128 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:01.792183 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:01.792202 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:01.792451 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:01.792614 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 11:26:01.792788 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":330,"threads_started":1}
I20260929 11:26:01.794775 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500: Bootstrap starting.
I20260929 11:26:01.797194 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:01.798137 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:01.799120 22413 log.cc:824] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:01.802838 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500: No bootstrap required, opened a new log
I20260929 11:26:01.805320 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:01.801416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58019 (local address 127.0.97.62:33415)
0929 11:26:01.801559 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:01.801563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:01.803406 (+ 1843us) server_negotiation.cc:407] Connection header received
0929 11:26:01.803598 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:01.803602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:01.803650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:01.803748 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:01.804654 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:01.804657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:01.804660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:01.804673 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:01.804731 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:01.804745 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:01.804883 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:01.805001 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 11:26:01.805066 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:26:01.811039 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16a942044c24986abce44ced1567500" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260929 11:26:01.811450 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:01.811563 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16a942044c24986abce44ced1567500, State: Initialized, Role: FOLLOWER
I20260929 11:26:01.812175 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16a942044c24986abce44ced1567500" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260929 11:26:01.812444 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:01.812625 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:01.812789 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:01.815245 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16a942044c24986abce44ced1567500" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260929 11:26:01.815737 22413 leader_election.cc:302] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e16a942044c24986abce44ced1567500; no voters:
I20260929 11:26:01.816329 22413 leader_election.cc:288] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:01.816457 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:01.817165 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [term 1 LEADER]: Becoming Leader. State: Replica: e16a942044c24986abce44ced1567500, State: Running, Role: LEADER
I20260929 11:26:01.817698 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16a942044c24986abce44ced1567500" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260929 11:26:01.818498 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:01.820694 22425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e16a942044c24986abce44ced1567500. Latest consensus state: current_term: 1 leader_uuid: "e16a942044c24986abce44ced1567500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16a942044c24986abce44ced1567500" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } } }
I20260929 11:26:01.820780 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e16a942044c24986abce44ced1567500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16a942044c24986abce44ced1567500" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } } }
I20260929 11:26:01.821178 22425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:01.821260 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:01.824128 22431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:01.829298 22431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:01.838424 22431 catalog_manager.cc:1347] Generated new cluster ID: 7b12ea7222d8473f8bf022abba16ec23
I20260929 11:26:01.838555 22431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:01.858783 22431 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:01.859258 22431 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:01.867062 22431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e16a942044c24986abce44ced1567500: Generated new TSK 0
I20260929 11:26:01.868008 22431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:01.878965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22346
W20260929 11:26:01.880088 22408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35951
W20260929 11:26:01.880813 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:01.880961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22346
2026-09-29T11:26:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:01 dist-test-slave-srst krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:01 dist-test-slave-srst krb5kdc[22444](info): set up 2 sockets
Sep 29 11:26:01 dist-test-slave-srst krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260929 11:26:02.965103 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:26:03 dist-test-slave-srst krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:26:03.209846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41897
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:03.529019 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:03.529372 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:03.529528 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:03.538044 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:03.538194 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:03.538269 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:03.538349 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:03.538414 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:03.548080 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:03.549433 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:03.550689 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:03.562424 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:03.562495 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:03.562515 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:03.563345 22457 server_base.cc:1034] running on GCE node
I20260929 11:26:03.564079 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:03.565516 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:03.566793 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790681163566749 us; error 50 us; skew 500 ppm
I20260929 11:26:03.567191 22457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:03 dist-test-slave-srst krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:03.570303 22457 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:26:03.571799 22457 webserver.cc:460] Webserver started at http://127.0.97.62:36055/ using document root <none> and password file <none>
I20260929 11:26:03.572489 22457 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:03.572670 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:03.572990 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:03.575778 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "0f14f79b6c474327956022d08f9777d2"
format_stamp: "Formatted at 2026-09-29 11:26:03 on dist-test-slave-srst"
server_key: "b8d93e385903afd238aa8fbe68ce1086"
server_key_iv: "c448aacc9fc31dc81325404ee07d0dc9"
server_key_version: "encryptionkey@0"
I20260929 11:26:03.576502 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f14f79b6c474327956022d08f9777d2"
format_stamp: "Formatted at 2026-09-29 11:26:03 on dist-test-slave-srst"
server_key: "b8d93e385903afd238aa8fbe68ce1086"
server_key_iv: "c448aacc9fc31dc81325404ee07d0dc9"
server_key_version: "encryptionkey@0"
I20260929 11:26:03.581516 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:26:03.584815 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:03.586086 22457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:26:03.586272 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "0f14f79b6c474327956022d08f9777d2"
format_stamp: "Formatted at 2026-09-29 11:26:03 on dist-test-slave-srst"
server_key: "b8d93e385903afd238aa8fbe68ce1086"
server_key_iv: "c448aacc9fc31dc81325404ee07d0dc9"
server_key_version: "encryptionkey@0"
I20260929 11:26:03.586467 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:03.622320 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:03.626323 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:03.626614 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:03.645117 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41897
I20260929 11:26:03.645124 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41897 every 8 connection(s)
I20260929 11:26:03.646497 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:03.650537 22524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:03.654390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22457
I20260929 11:26:03.654587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:26:03.655019 388 external_mini_cluster.cc:1428] Setting key 92f31412732985f81280a59442e43aac
I20260929 11:26:03.663576 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2: Bootstrap starting.
I20260929 11:26:03.667472 22524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:03.668862 22524 log.cc:824] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:26:03 dist-test-slave-srst krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:26:03.671967 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2: No bootstrap required, opened a new log
I20260929 11:26:03.679953 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:03.656916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57171 (local address 127.0.97.62:41897)
0929 11:26:03.657716 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:03.657727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:03.657788 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:26:03.659344 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:03.659366 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:03.659789 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:03.660113 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:03.661889 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:03.663270 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:03.665225 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:03.665896 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:03.666023 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:03.671224 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:03.671232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:03.671247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:03.671291 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:03.675269 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:03.675952 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:03.675964 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:03.675977 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:03.676208 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:03.676630 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:03.676635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:03.676637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:03.677339 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:03.677686 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:03.678111 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 11:26:03.678550 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":400,"threads_started":1}
I20260929 11:26:03.682184 22524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f14f79b6c474327956022d08f9777d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41897 } }
I20260929 11:26:03.682570 22524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:03.682691 22524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f14f79b6c474327956022d08f9777d2, State: Initialized, Role: FOLLOWER
I20260929 11:26:03.683300 22524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f14f79b6c474327956022d08f9777d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41897 } }
I20260929 11:26:03.683555 22524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:03.683700 22524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:03.683876 22524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:03.683940 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:03.686057 22524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f14f79b6c474327956022d08f9777d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41897 } }
I20260929 11:26:03.686532 22524 leader_election.cc:302] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f14f79b6c474327956022d08f9777d2; no voters:
I20260929 11:26:03.687198 22524 leader_election.cc:288] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:03.687288 22532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:03.689782 22524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:03.692461 22532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [term 1 LEADER]: Becoming Leader. State: Replica: 0f14f79b6c474327956022d08f9777d2, State: Running, Role: LEADER
I20260929 11:26:03.693189 22532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f14f79b6c474327956022d08f9777d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41897 } }
I20260929 11:26:03.696902 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f14f79b6c474327956022d08f9777d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f14f79b6c474327956022d08f9777d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41897 } } }
I20260929 11:26:03.697284 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:03.698401 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f14f79b6c474327956022d08f9777d2. Latest consensus state: current_term: 1 leader_uuid: "0f14f79b6c474327956022d08f9777d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f14f79b6c474327956022d08f9777d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41897 } } }
I20260929 11:26:03.698732 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:03.704262 22542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:03.710002 22542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:03.712702 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:03.688775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44925 (local address 127.0.97.62:41897)
0929 11:26:03.688933 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:03.688939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:03.692576 (+ 3637us) server_negotiation.cc:407] Connection header received
0929 11:26:03.692821 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:03.692831 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:03.692959 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:03.693109 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:03.695022 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:03.696832 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:03.699487 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:03.700360 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:03.700498 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:03.704898 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:03.704906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:03.704911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:03.704949 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:03.708339 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:03.709264 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:03.709272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:03.709276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:03.709434 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:03.709903 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:03.709907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:03.709909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:03.710474 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:03.710870 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:03.711690 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 11:26:03.712011 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260929 11:26:03.720142 22542 catalog_manager.cc:1347] Generated new cluster ID: 58a2d68a46e7480c983e12f3b2266fdb
I20260929 11:26:03.720288 22542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:03.741221 22542 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:03.743172 22542 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:03.764479 22542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f14f79b6c474327956022d08f9777d2: Generated new TSK 0
I20260929 11:26:03.765448 22542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:03.781389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22457
I20260929 11:26:03.782687 22519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:03.875392 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22457
2026-09-29T11:26:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:03 dist-test-slave-srst krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:03 dist-test-slave-srst krb5kdc[22556](info): set up 2 sockets
Sep 29 11:26:03 dist-test-slave-srst krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260929 11:26:04.981364 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:26:05 dist-test-slave-srst krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:26:05.228053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35463
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:05.552064 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:05.552363 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:05.552456 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:05.560246 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:05.560355 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:05.560406 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:05.560463 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:05.560513 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:05.570678 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:05.571877 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:05.573170 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:05.584336 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:05.584299 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:05.584441 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:05.585469 22569 server_base.cc:1034] running on GCE node
I20260929 11:26:05.586210 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:05.587545 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:05.588759 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790681165588717 us; error 55 us; skew 500 ppm
I20260929 11:26:05.589105 22569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:05 dist-test-slave-srst krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:05.592218 22569 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:26:05.593796 22569 webserver.cc:460] Webserver started at http://127.0.97.62:41019/ using document root <none> and password file <none>
I20260929 11:26:05.594489 22569 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:05.594638 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:05.594940 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:05.597954 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "4c91aba489fc451c9b874cfcda417d19"
format_stamp: "Formatted at 2026-09-29 11:26:05 on dist-test-slave-srst"
server_key: "861a3ec0ed2a9db1e839670335e47230"
server_key_iv: "ccf9085564eabc594c14f58a329d5ece"
server_key_version: "encryptionkey@0"
I20260929 11:26:05.598699 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c91aba489fc451c9b874cfcda417d19"
format_stamp: "Formatted at 2026-09-29 11:26:05 on dist-test-slave-srst"
server_key: "861a3ec0ed2a9db1e839670335e47230"
server_key_iv: "ccf9085564eabc594c14f58a329d5ece"
server_key_version: "encryptionkey@0"
I20260929 11:26:05.603600 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 11:26:05.606953 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:05.608232 22569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:26:05.608415 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "4c91aba489fc451c9b874cfcda417d19"
format_stamp: "Formatted at 2026-09-29 11:26:05 on dist-test-slave-srst"
server_key: "861a3ec0ed2a9db1e839670335e47230"
server_key_iv: "ccf9085564eabc594c14f58a329d5ece"
server_key_version: "encryptionkey@0"
I20260929 11:26:05.608649 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:05.633458 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:05.637957 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:05.638252 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:05.656266 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35463
I20260929 11:26:05.656296 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35463 every 8 connection(s)
I20260929 11:26:05.657706 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:05.661480 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:05.661939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22569
I20260929 11:26:05.662120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:26:05.662664 388 external_mini_cluster.cc:1428] Setting key ac3014eac700b79bc2134d291fce581a
I20260929 11:26:05.675307 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19: Bootstrap starting.
I20260929 11:26:05.676448 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:05.664400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59509 (local address 127.0.97.62:35463)
0929 11:26:05.665291 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:05.665302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:05.665339 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:26:05.667694 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:05.667720 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:05.668188 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:05.668524 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:05.669113 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:05.670319 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:05.672203 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:05.672985 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:05.673123 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:05.674332 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:05.674340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:05.674351 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:05.674851 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:05.674921 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:05.674944 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:05.675028 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:05.675257 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 11:26:05.675512 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":459,"threads_started":1}
I20260929 11:26:05.678445 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:05.679630 22636 log.cc:824] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:05.679992 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:05.682924 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19: No bootstrap required, opened a new log
I20260929 11:26:05.691921 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:05.685296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41353 (local address 127.0.97.62:35463)
0929 11:26:05.685517 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:05.685524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:05.685871 (+ 347us) server_negotiation.cc:407] Connection header received
0929 11:26:05.686091 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:05.686095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:05.686158 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:05.686234 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:05.686845 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:05.687560 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:05.689475 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:05.689930 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:05.690021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:05.691167 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:05.691172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:05.691178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:05.691194 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:05.691235 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:05.691252 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:05.691316 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:05.691468 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 11:26:05.691535 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:26:05.693652 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c91aba489fc451c9b874cfcda417d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35463 } }
I20260929 11:26:05.694051 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:05.694169 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c91aba489fc451c9b874cfcda417d19, State: Initialized, Role: FOLLOWER
I20260929 11:26:05.694763 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c91aba489fc451c9b874cfcda417d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35463 } }
I20260929 11:26:05.694998 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:05.695147 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:05.695327 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:05.697587 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c91aba489fc451c9b874cfcda417d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35463 } }
I20260929 11:26:05.698099 22636 leader_election.cc:302] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c91aba489fc451c9b874cfcda417d19; no voters:
I20260929 11:26:05.698693 22636 leader_election.cc:288] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:05.698829 22646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:05.700155 22646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [term 1 LEADER]: Becoming Leader. State: Replica: 4c91aba489fc451c9b874cfcda417d19, State: Running, Role: LEADER
I20260929 11:26:05.700704 22646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c91aba489fc451c9b874cfcda417d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35463 } }
I20260929 11:26:05.701139 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:05.703665 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c91aba489fc451c9b874cfcda417d19. Latest consensus state: current_term: 1 leader_uuid: "4c91aba489fc451c9b874cfcda417d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c91aba489fc451c9b874cfcda417d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35463 } } }
I20260929 11:26:05.704105 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:05.703639 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c91aba489fc451c9b874cfcda417d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c91aba489fc451c9b874cfcda417d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35463 } } }
I20260929 11:26:05.704466 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:05.705526 22655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:05.711282 22655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:05.719849 22655 catalog_manager.cc:1347] Generated new cluster ID: 0ecd9ccb7f6e4e7abf15bb54b0d39946
I20260929 11:26:05.719950 22655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:05.741138 22655 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:05.742527 22655 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:05.754441 22655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c91aba489fc451c9b874cfcda417d19: Generated new TSK 0
I20260929 11:26:05.755470 22655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:05.768870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22569
W20260929 11:26:05.770308 22631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59509
W20260929 11:26:05.770913 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:05.771064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22569
2026-09-29T11:26:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:05 dist-test-slave-srst krb5kdc[22667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:05 dist-test-slave-srst krb5kdc[22667](info): set up 2 sockets
Sep 29 11:26:05 dist-test-slave-srst krb5kdc[22667](info): commencing operation
krb5kdc: starting...
W20260929 11:26:06.857075 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.010s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:26:07 dist-test-slave-srst krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:26:07.104718 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36111
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:07.425230 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:07.425596 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:07.425761 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:07.432982 22680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:07.433125 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:07.433234 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:07.433334 22680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:07.433418 22680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:07.443158 22680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:07.444453 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:07.445778 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:07.457149 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:07.457232 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:07.457190 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:07.458055 22680 server_base.cc:1034] running on GCE node
I20260929 11:26:07.458724 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:07.460002 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:07.461200 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790681167461166 us; error 29 us; skew 500 ppm
I20260929 11:26:07.461499 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:07 dist-test-slave-srst krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:07.471868 22680 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:26:07.473593 22680 webserver.cc:460] Webserver started at http://127.0.97.62:35549/ using document root <none> and password file <none>
I20260929 11:26:07.474359 22680 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:07.474532 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:07.474879 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:07.478276 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc"
format_stamp: "Formatted at 2026-09-29 11:26:07 on dist-test-slave-srst"
server_key: "14b253ac6abbbe2c2af2e1270d13476b"
server_key_iv: "a1765b1e5b4660d84edf149ebdc064dc"
server_key_version: "encryptionkey@0"
I20260929 11:26:07.479130 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc"
format_stamp: "Formatted at 2026-09-29 11:26:07 on dist-test-slave-srst"
server_key: "14b253ac6abbbe2c2af2e1270d13476b"
server_key_iv: "a1765b1e5b4660d84edf149ebdc064dc"
server_key_version: "encryptionkey@0"
I20260929 11:26:07.484737 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 11:26:07.488278 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:07.489660 22680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:07.489872 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc"
format_stamp: "Formatted at 2026-09-29 11:26:07 on dist-test-slave-srst"
server_key: "14b253ac6abbbe2c2af2e1270d13476b"
server_key_iv: "a1765b1e5b4660d84edf149ebdc064dc"
server_key_version: "encryptionkey@0"
I20260929 11:26:07.490067 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:07.496085 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:07.500774 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:07.501104 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:07.520517 22680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36111
I20260929 11:26:07.520550 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36111 every 8 connection(s)
I20260929 11:26:07.521968 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:07.525655 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:07.528182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22680
I20260929 11:26:07.528383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:26:07.528901 388 external_mini_cluster.cc:1428] Setting key 3e9879864091940600d8cb0d27396d41
I20260929 11:26:07.536713 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:07.530771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52237 (local address 127.0.97.62:36111)
0929 11:26:07.531562 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:07.531572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:07.531613 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:26:07.533283 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:07.533305 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:07.533633 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:07.533957 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:07.535047 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:07.535053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:07.535071 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:07.535102 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:07.535168 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:07.535198 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:07.535392 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:07.535508 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 11:26:07.535710 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":366,"threads_started":1}
I20260929 11:26:07.537446 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc: Bootstrap starting.
I20260929 11:26:07.540421 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:07.540820 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:07.541780 22747 log.cc:824] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:07.544718 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc: No bootstrap required, opened a new log
I20260929 11:26:07.548182 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:07.545383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55573 (local address 127.0.97.62:36111)
0929 11:26:07.545578 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:07.545586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:07.546020 (+ 434us) server_negotiation.cc:407] Connection header received
0929 11:26:07.546244 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:07.546249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:07.546295 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:07.546414 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:07.547387 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:07.547393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:07.547402 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:07.547423 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:07.547474 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:07.547490 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:07.547623 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:07.547738 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 11:26:07.547800 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:26:07.553546 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } }
I20260929 11:26:07.553927 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:07.554023 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4889dcf5f642cc9d63c2ae4deecdcc, State: Initialized, Role: FOLLOWER
I20260929 11:26:07.554782 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } }
I20260929 11:26:07.555012 22757 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:07.555279 22757 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } }
I20260929 11:26:07.555960 22757 leader_election.cc:302] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd4889dcf5f642cc9d63c2ae4deecdcc; no voters:
I20260929 11:26:07.556726 22747 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:07.556845 22757 leader_election.cc:288] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260929 11:26:07.556914 22747 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:07.557026 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:07.559206 22747 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } }
I20260929 11:26:07.559466 22747 leader_election.cc:302] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd4889dcf5f642cc9d63c2ae4deecdcc; no voters:
I20260929 11:26:07.559525 22758 raft_consensus.cc:2741] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 11:26:07.559759 22747 leader_election.cc:288] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:07.559873 22757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:07.560691 22757 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [term 1 LEADER]: Becoming Leader. State: Replica: cd4889dcf5f642cc9d63c2ae4deecdcc, State: Running, Role: LEADER
I20260929 11:26:07.561236 22757 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } }
I20260929 11:26:07.562146 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:07.563184 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } } }
I20260929 11:26:07.563182 22760 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd4889dcf5f642cc9d63c2ae4deecdcc. Latest consensus state: current_term: 1 leader_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd4889dcf5f642cc9d63c2ae4deecdcc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36111 } } }
I20260929 11:26:07.563707 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:07.563683 22760 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:07.564539 22763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:07.572118 22763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:07.580866 22763 catalog_manager.cc:1347] Generated new cluster ID: e7cb7c6d7c3049f794d0bdfe37a7b5e9
I20260929 11:26:07.580981 22763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:07.601081 22763 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:07.601593 22763 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:07.613062 22763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd4889dcf5f642cc9d63c2ae4deecdcc: Generated new TSK 0
I20260929 11:26:07.613843 22763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:07.626307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22680
W20260929 11:26:07.627391 22742 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52237
W20260929 11:26:07.628054 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:07.628203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22680
2026-09-29T11:26:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1860 ms)
[----------] 18 tests from AuthTokenIssuingTest (34506 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:07 dist-test-slave-srst krb5kdc[22779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:07 dist-test-slave-srst krb5kdc[22779](info): set up 2 sockets
Sep 29 11:26:07 dist-test-slave-srst krb5kdc[22779](info): commencing operation
krb5kdc: starting...
W20260929 11:26:08.741220 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:26:08 dist-test-slave-srst krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:26:08.988029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39117
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39117,127.0.97.61:45111,127.0.97.60:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:09.314656 22792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:26:09.314990 22792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:09.315088 22792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:09.315263 22792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:09.322562 22792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:09.322654 22792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:09.322711 22792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:09.322788 22792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:09.322893 22792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:09.333058 22792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39117,127.0.97.61:45111,127.0.97.60:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:09.334429 22792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:09.335762 22792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:09.347939 22797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:09.347890 22800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:09.347965 22798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:09.349128 22792 server_base.cc:1034] running on GCE node
I20260929 11:26:09.349923 22792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:09.351418 22792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:09.352705 22792 hybrid_clock.cc:648] HybridClock initialized: now 1790681169352624 us; error 47 us; skew 500 ppm
I20260929 11:26:09.353053 22792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:09 dist-test-slave-srst krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:09.356278 22792 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:26:09.357926 22792 webserver.cc:460] Webserver started at http://127.0.97.62:42687/ using document root <none> and password file <none>
I20260929 11:26:09.358654 22792 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:09.358790 22792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:09.359115 22792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:09.362197 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "b050af65c3fd4b1eb45acb2086f68711"
format_stamp: "Formatted at 2026-09-29 11:26:09 on dist-test-slave-srst"
server_key: "71484905d4420cbc17524eb4c9895d39"
server_key_iv: "d0ed888be1ace743b611a8cb2bf8d51e"
server_key_version: "encryptionkey@0"
I20260929 11:26:09.362963 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "b050af65c3fd4b1eb45acb2086f68711"
format_stamp: "Formatted at 2026-09-29 11:26:09 on dist-test-slave-srst"
server_key: "71484905d4420cbc17524eb4c9895d39"
server_key_iv: "d0ed888be1ace743b611a8cb2bf8d51e"
server_key_version: "encryptionkey@0"
I20260929 11:26:09.368186 22792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:26:09.371675 22807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:09.373030 22792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:09.373211 22792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "b050af65c3fd4b1eb45acb2086f68711"
format_stamp: "Formatted at 2026-09-29 11:26:09 on dist-test-slave-srst"
server_key: "71484905d4420cbc17524eb4c9895d39"
server_key_iv: "d0ed888be1ace743b611a8cb2bf8d51e"
server_key_version: "encryptionkey@0"
I20260929 11:26:09.373430 22792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:09.386952 22792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:09.391049 22792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:09.391358 22792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:09.410382 22792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39117
I20260929 11:26:09.410388 22858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39117 every 8 connection(s)
I20260929 11:26:09.411832 22792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:09.415644 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:09.421136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22792
I20260929 11:26:09.421350 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:26:09.422122 388 external_mini_cluster.cc:1428] Setting key 5b62632ffe6826963d78649ee3a37713
I20260929 11:26:09.423285 22859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:09 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:26:09.448974 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.427304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39117 (local address 127.0.0.1:44334)
0929 11:26:09.428246 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.428316 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:09.431244 (+ 2928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:09.432214 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:09.432221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:09.432958 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:09.433305 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:09.433322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.434828 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.434834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:09.435928 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:09.435939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.436957 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.436963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:09.437147 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:09.437829 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:09.437851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:09.440261 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:09.444885 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:09.444895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:09.444905 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:09.445295 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:09.445800 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:09.445804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:09.445806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:09.445952 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:09.446995 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:09.447001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:09.447196 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 11:26:09.447405 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 11:26:09.447810 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":386,"threads_started":1}
I20260929 11:26:09.449014 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.427304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.0.97.62:39117)
0929 11:26:09.429266 (+ 1962us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:09.429272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:09.429318 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:26:09.431737 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:09.431742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:09.431981 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:09.432207 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:09.433517 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.434677 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.436103 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.436728 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.436941 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:09.440523 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:09.440537 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:09.440556 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:09.440619 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:09.444747 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:09.445441 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:09.445453 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:09.445460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:09.445675 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:09.446066 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:09.446075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:09.446080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:09.446858 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:09.447082 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:09.447598 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 11:26:09.448011 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":1771,"spinlock_wait_cycles":36608,"thread_start_us":1549,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260929 11:26:09.451960 22859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 11:26:09.453315 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.452816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59120)
0929 11:26:09.453019 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.453161 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 11:26:09.453615 22811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
I20260929 11:26:09.455230 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.454786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59122)
0929 11:26:09.454962 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.455054 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:26:09.455430 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 11:26:09.483397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45111
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39117,127.0.97.61:45111,127.0.97.60:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:26:09.504770 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } attempt: 1
I20260929 11:26:09.505918 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.505458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59138)
0929 11:26:09.505681 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.505775 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 11:26:09.506901 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.506563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59152)
0929 11:26:09.506701 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.506771 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:26:09.507104 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
I20260929 11:26:09.575389 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } attempt: 2
I20260929 11:26:09.576690 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.576149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59168)
0929 11:26:09.576407 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.576494 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 11:26:09.577782 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.577454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59172)
0929 11:26:09.577580 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.577660 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:26:09.577961 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
I20260929 11:26:09.669322 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } attempt: 3
I20260929 11:26:09.670447 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.669995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59186)
0929 11:26:09.670201 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.670300 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 11:26:09.671396 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.671065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59188)
0929 11:26:09.671198 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.671269 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:26:09.671591 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
I20260929 11:26:09.814831 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } attempt: 4
I20260929 11:26:09.816028 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.815494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59202)
0929 11:26:09.815797 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.815885 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260929 11:26:09.815887 22866 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:26:09.816349 22866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:09.816437 22866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:09.816623 22866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 11:26:09.817296 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.816839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59212)
0929 11:26:09.817010 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.817144 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:26:09.817545 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45111: connect: Connection refused (error 111)
W20260929 11:26:09.825177 22866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:09.825326 22866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:09.825438 22866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:09.825525 22866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:09.825635 22866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:09.836074 22866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39117,127.0.97.61:45111,127.0.97.60:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:09.837603 22866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:09.839002 22866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:09.850594 22872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:09.850708 22874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:09.850736 22871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:09.851462 22866 server_base.cc:1034] running on GCE node
I20260929 11:26:09.852252 22866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:09.853643 22866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:09.854887 22866 hybrid_clock.cc:648] HybridClock initialized: now 1790681169854834 us; error 33 us; skew 500 ppm
I20260929 11:26:09.855211 22866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:09 dist-test-slave-srst krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:09.858332 22866 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 11:26:09.859712 22866 webserver.cc:460] Webserver started at http://127.0.97.61:46295/ using document root <none> and password file <none>
I20260929 11:26:09.860392 22866 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:09.860519 22866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:09.860903 22866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:09.863801 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data/instance:
uuid: "74e39a3c684643f29afaf14a05f8e79f"
format_stamp: "Formatted at 2026-09-29 11:26:09 on dist-test-slave-srst"
server_key: "15879a4785510d6a97c97e8c1069253f"
server_key_iv: "042bfb667fffaa9fb44505a4f468e41e"
server_key_version: "encryptionkey@0"
I20260929 11:26:09.864677 22866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal/instance:
uuid: "74e39a3c684643f29afaf14a05f8e79f"
format_stamp: "Formatted at 2026-09-29 11:26:09 on dist-test-slave-srst"
server_key: "15879a4785510d6a97c97e8c1069253f"
server_key_iv: "042bfb667fffaa9fb44505a4f468e41e"
server_key_version: "encryptionkey@0"
I20260929 11:26:09.869599 22866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:26:09.872889 22881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:09.874137 22866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:09.874316 22866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal
uuid: "74e39a3c684643f29afaf14a05f8e79f"
format_stamp: "Formatted at 2026-09-29 11:26:09 on dist-test-slave-srst"
server_key: "15879a4785510d6a97c97e8c1069253f"
server_key_iv: "042bfb667fffaa9fb44505a4f468e41e"
server_key_version: "encryptionkey@0"
I20260929 11:26:09.874531 22866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:09.894222 22866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:09.898092 22866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:09.898353 22866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:09.916457 22866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45111
I20260929 11:26:09.916483 22932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45111 every 8 connection(s)
I20260929 11:26:09.917873 22866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
I20260929 11:26:09.919188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22866
I20260929 11:26:09.919394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-1/wal/instance
I20260929 11:26:09.919921 388 external_mini_cluster.cc:1428] Setting key 3fadb06daf7b2740bde354a63a430f15
I20260929 11:26:09.922528 22933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:09.932168 22933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Sep 29 11:26:09 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 11:26:09.953739 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.935142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.97.62:39117)
0929 11:26:09.935332 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:09.935337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:09.936677 (+ 1340us) server_negotiation.cc:407] Connection header received
0929 11:26:09.938708 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:09.938714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:09.938814 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:09.938930 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:09.940518 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.941260 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.942951 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.943429 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.943573 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:09.947167 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:09.947173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:09.947177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:09.947208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:09.950462 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:09.951191 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:09.951199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:09.951205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:09.951358 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:09.951748 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:09.951755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:09.951761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:09.952152 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:09.952305 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:09.953048 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 11:26:09.953303 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:26:09.954471 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.935645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39117 (local address 127.0.0.1:44340)
0929 11:26:09.936473 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.936527 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:09.938398 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:09.939187 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:09.939196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:09.939745 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:09.940331 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:09.940349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.941431 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.941435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:09.942804 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:09.942815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.943670 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.943677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:09.943913 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:09.944704 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:09.944727 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:09.946920 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:09.950630 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:09.950637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:09.950650 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:09.951063 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:09.951469 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:09.951472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:09.951475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:09.951636 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:09.952270 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:09.952275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:09.952670 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 11:26:09.952911 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 11:26:09.953286 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
I20260929 11:26:09.956382 22933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:09 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:26:09.972635 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.957262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59220)
0929 11:26:09.957481 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.957509 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:09.957619 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:09.958504 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:09.958507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:09.958902 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:09.959031 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:09.959038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.961120 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.961123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:09.961946 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:09.961956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.963164 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.963168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:09.963301 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:09.963934 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:09.963950 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:09.965989 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:09.970246 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:09.970250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:09.970252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:09.970610 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:09.971021 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:09.971024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:09.971026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:09.971159 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:09.971895 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:09.971898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:09.972015 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 11:26:09.972124 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 11:26:09.972313 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":1,"spinlock_wait_cycles":17280}
I20260929 11:26:09.973173 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.957345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.0.97.61:45111)
0929 11:26:09.957960 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:09.957966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:09.958017 (+ 51us) server_negotiation.cc:407] Connection header received
0929 11:26:09.958077 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:09.958087 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:09.958335 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:09.958496 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:09.959202 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.960975 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.962084 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:09.962999 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:09.963149 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:09.966205 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:09.966221 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:09.966241 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:09.966281 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:09.970122 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:09.970741 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:09.970752 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:09.970759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:09.970918 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:09.971262 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:09.971270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:09.971276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:09.971752 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:09.971984 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:09.972319 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 11:26:09.972639 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":469,"thread_start_us":329,"threads_started":1}
I20260929 11:26:09.975658 22933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:26:09.977015 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.976433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47924)
0929 11:26:09.976704 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.976838 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 11:26:09.977380 22884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:09.979116 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:09.978773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47936)
0929 11:26:09.978919 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:09.978995 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 11:26:09.979346 22933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 11:26:09.983762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44637
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39117,127.0.97.61:45111,127.0.97.60:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:26:10.028795 22933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 1
I20260929 11:26:10.029999 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.029570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47952)
0929 11:26:10.029756 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.029850 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 11:26:10.030997 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.030645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47962)
0929 11:26:10.030820 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.030895 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 11:26:10.031201 22933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:10.099527 22933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 2
I20260929 11:26:10.100710 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.100196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47968)
0929 11:26:10.100403 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.100498 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 11:26:10.101712 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.101381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47978)
0929 11:26:10.101535 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.101596 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:26:10.101912 22933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:10.116812 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } attempt: 5
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:26:10.130630 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.117634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:59232)
0929 11:26:10.117827 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.117850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:10.117996 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:10.118387 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:10.118395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:10.118806 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:10.118991 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.119000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.120005 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.120010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.120935 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.120947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.121696 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.121701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.121985 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.122774 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:10.122797 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:10.125147 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:10.128188 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.128194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.128198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.128563 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.128950 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.128962 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.128966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.129105 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.129619 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:10.129624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:10.129782 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 11:26:10.129947 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 11:26:10.130255 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 11:26:10.130625 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.117755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.0.97.61:45111)
0929 11:26:10.117958 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.117964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.117985 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:26:10.118150 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.118161 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.118237 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.118347 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.119153 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.119851 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.121092 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.121537 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.121667 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.125353 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.125364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.125367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.125398 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.128051 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.128715 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.128724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.128726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.128839 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.129219 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.129223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.129228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.129502 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.129652 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.130077 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.130282 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:26:10.132063 22859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:26:10.133126 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.132748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47992)
0929 11:26:10.132912 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.132990 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:26:10.134141 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.133761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:47994)
0929 11:26:10.133923 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.134003 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 11:26:10.134343 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:10.185580 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 1
I20260929 11:26:10.186865 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.186281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48000)
0929 11:26:10.186528 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.186644 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 11:26:10.187877 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.187497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48006)
0929 11:26:10.187656 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.187738 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 11:26:10.188102 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:10.193121 22933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 3
I20260929 11:26:10.194139 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.193722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48018)
0929 11:26:10.193908 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.193998 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 11:26:10.195238 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.194866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48026)
0929 11:26:10.195075 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.195147 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260929 11:26:10.195425 22933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:10.256347 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 2
I20260929 11:26:10.257619 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.257026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48042)
0929 11:26:10.257268 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.257405 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 11:26:10.258775 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.258345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48054)
0929 11:26:10.258525 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.258616 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 11:26:10.259039 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
W20260929 11:26:10.315743 22940 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:26:10.316040 22940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:10.316121 22940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:10.316197 22940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:10.323020 22940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:10.323100 22940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:10.323155 22940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:10.323205 22940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:10.323258 22940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:10.332280 22940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39117,127.0.97.61:45111,127.0.97.60:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:10.333446 22940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:10.334630 22940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:26:10.338617 22933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 4
I20260929 11:26:10.339840 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.339313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48062)
0929 11:26:10.339559 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.339687 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 11:26:10.341143 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.340690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48066)
0929 11:26:10.340923 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.341013 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 11:26:10.341394 22933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
W20260929 11:26:10.346379 22948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:10.346452 22946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:10.346886 22945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:10.347407 22940 server_base.cc:1034] running on GCE node
I20260929 11:26:10.348178 22940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:10.349579 22940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:10.350818 22940 hybrid_clock.cc:648] HybridClock initialized: now 1790681170350752 us; error 40 us; skew 500 ppm
I20260929 11:26:10.351140 22940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:10.354208 22940 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 11:26:10.355544 22940 webserver.cc:460] Webserver started at http://127.0.97.60:38225/ using document root <none> and password file <none>
I20260929 11:26:10.356225 22940 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:10.356362 22940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:10.356737 22940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:10.359634 22940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data/instance:
uuid: "a0747aafb4f9400891dac3d06bd90128"
format_stamp: "Formatted at 2026-09-29 11:26:10 on dist-test-slave-srst"
server_key: "9accb54969ec6a97ca387cd6d554d57d"
server_key_iv: "306e9f6e6dcf8632e12ba4891f7057f2"
server_key_version: "encryptionkey@0"
I20260929 11:26:10.360407 22940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal/instance:
uuid: "a0747aafb4f9400891dac3d06bd90128"
format_stamp: "Formatted at 2026-09-29 11:26:10 on dist-test-slave-srst"
server_key: "9accb54969ec6a97ca387cd6d554d57d"
server_key_iv: "306e9f6e6dcf8632e12ba4891f7057f2"
server_key_version: "encryptionkey@0"
I20260929 11:26:10.365262 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 3
I20260929 11:26:10.365458 22940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:26:10.366365 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.365857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48078)
0929 11:26:10.366095 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.366201 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 11:26:10.367534 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.367027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:48080)
0929 11:26:10.367303 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.367396 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260929 11:26:10.367810 22859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44637: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44637: connect: Connection refused (error 111)
I20260929 11:26:10.368934 22955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:10.370160 22940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:10.370342 22940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal
uuid: "a0747aafb4f9400891dac3d06bd90128"
format_stamp: "Formatted at 2026-09-29 11:26:10 on dist-test-slave-srst"
server_key: "9accb54969ec6a97ca387cd6d554d57d"
server_key_iv: "306e9f6e6dcf8632e12ba4891f7057f2"
server_key_version: "encryptionkey@0"
I20260929 11:26:10.370543 22940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:10.391131 22940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:10.395234 22940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:10.395526 22940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:10.414438 22940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44637
I20260929 11:26:10.414465 23006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44637 every 8 connection(s)
I20260929 11:26:10.415735 22940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
I20260929 11:26:10.418352 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 22940
I20260929 11:26:10.418546 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790680821855275-388-0/minicluster-data/master-2/wal/instance
I20260929 11:26:10.418967 388 external_mini_cluster.cc:1428] Setting key b0e69f6343c640bde01256fcff7eff57
I20260929 11:26:10.420025 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:10.429111 23007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:26:10.441964 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.420800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.97.60:44637)
0929 11:26:10.421548 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.421559 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.421598 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:26:10.423067 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.423086 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.423456 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.423786 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.425532 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.426765 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.428546 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.429203 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.429329 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.433900 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.433909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.433921 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.433966 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.437562 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.438420 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.438428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.438435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.438662 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.439127 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.439131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.439134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.439856 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.440175 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.440485 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.440905 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":363,"threads_started":1}
I20260929 11:26:10.447888 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.432021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.0.97.62:39117)
0929 11:26:10.432257 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.432265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.432919 (+ 654us) server_negotiation.cc:407] Connection header received
0929 11:26:10.433076 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.433082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.433161 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.433290 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.434515 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.435356 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.437411 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.438038 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.438195 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.441799 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.441804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.441807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.441835 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.444858 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.445788 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.445794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.445798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.445986 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.446386 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.446391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.446396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.446743 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.446928 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.447424 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.447615 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:26:10.448032 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.432007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39117 (local address 127.0.0.1:60670)
0929 11:26:10.432778 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.432817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:10.432957 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:10.433334 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:10.433338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:10.434026 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:10.434371 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.434382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.435517 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.435522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.437258 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.437274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.438205 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.438209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.438369 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.439062 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:10.439089 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:10.441602 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:10.445047 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.445059 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.445067 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.445634 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.446113 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.446118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.446121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.446263 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.446895 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:10.446899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:10.447097 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 11:26:10.447288 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 11:26:10.447633 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":389,"threads_started":1}
I20260929 11:26:10.449499 23007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:26:10.460119 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.444712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.0.97.62:39117)
0929 11:26:10.445334 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.445339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.445361 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:26:10.445409 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.445414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.445485 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.445595 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.447033 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.448303 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.449868 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.450393 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.450516 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.454817 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.454825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.454831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.454867 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.457590 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.458183 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.458190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.458199 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.458315 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.458671 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.458676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.458681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.458940 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.459249 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.459554 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.459781 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":237,"threads_started":1}
I20260929 11:26:10.462589 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.450188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45111 (local address 127.0.0.1:54750)
0929 11:26:10.450341 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.450360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:10.450443 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:10.450813 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:10.450819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:10.451270 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:10.451452 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.451464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.452362 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.452367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.453559 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.453570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.454481 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.454485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.454617 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.455120 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:10.455134 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:10.457080 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:10.459994 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.459998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.460002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.460436 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.460865 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.460869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.460872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.461048 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.461674 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:10.461679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:10.461838 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 11:26:10.461981 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 11:26:10.462244 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:26:10.462666 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.450300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.0.97.61:45111)
0929 11:26:10.450495 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.450501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.450519 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:26:10.450578 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.450583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.450679 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.450763 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.451596 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.452212 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.453690 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.454348 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.454481 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.457264 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.457270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.457273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.457303 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.459815 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.460558 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.460563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.460566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.460744 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.461169 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.461174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.461177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.461543 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.461687 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.462074 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.462284 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:26:10.464131 23007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 11:26:10.478011 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.461887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.0.97.61:45111)
0929 11:26:10.462903 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.462909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.462926 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:26:10.463016 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.463022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.463091 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.463178 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.464714 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.465587 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.467060 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.467424 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.467510 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.471988 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.471994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.471999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.472033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.475017 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.475755 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.475770 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.475775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.475912 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.476380 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.476386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.476389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.476732 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.476960 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.477332 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.477621 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":1438,"threads_started":1}
I20260929 11:26:10.478012 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.464956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:33300)
0929 11:26:10.465769 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.465782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:10.465851 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:10.466114 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:10.466117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:10.466425 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:10.466541 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.466548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.467642 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.467649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.468659 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.468670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.469652 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.469656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.469790 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.470306 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:10.470321 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:10.472439 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:10.475491 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.475494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.475497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.475911 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.476324 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.476329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.476334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.476495 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.477080 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:10.477083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:10.477246 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 11:26:10.477408 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 11:26:10.477654 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723}
I20260929 11:26:10.478866 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.464927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.97.60:44637)
0929 11:26:10.465098 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.465103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.465846 (+ 743us) server_negotiation.cc:407] Connection header received
0929 11:26:10.465922 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.465927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.465996 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.466093 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.466681 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.467495 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.468804 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.469509 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.469648 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.472650 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.472658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.472666 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.472692 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.475378 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.476028 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.476033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.476036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.476177 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.476650 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.476653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.476655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.476965 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.477130 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.478309 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.478513 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:26:10.483040 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:10.483997 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: Bootstrap starting.
I20260929 11:26:10.487258 23007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:10.485684 22834 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b050af65c3fd4b1eb45acb2086f68711"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676
I20260929 11:26:10.488300 23007 log.cc:824] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:10.490978 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: No bootstrap required, opened a new log
I20260929 11:26:10.497828 23007 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.498137 23007 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0747aafb4f9400891dac3d06bd90128, State: Initialized, Role: FOLLOWER
I20260929 11:26:10.498560 23007 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.499501 23015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.499889 23015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.500626 23007 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:26:10.509698 23026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:26:10.509827 23026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:26:10.507813 22940 master_runner.cc:186] Error getting master registration for 127.0.97.62:39117: OK, instance_id { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" instance_seqno: 1790681169405074 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:26:10.515460 22940 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:26:10.545045 22859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 4
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 11:26:10.559964 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.545822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:33306)
0929 11:26:10.546032 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.546065 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:10.546240 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:10.546592 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:10.546596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:10.546990 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:10.547154 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.547163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.548165 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.548173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.549158 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.549169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.549925 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.549931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.550122 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.550715 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:10.550732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:10.552916 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:10.557058 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.557066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.557072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.557593 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.558006 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.558011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.558014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.558180 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.558930 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:10.558935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:10.559099 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 11:26:10.559253 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 11:26:10.559557 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:26:10.560014 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.545852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.0.97.60:44637)
0929 11:26:10.546072 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.546078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.546165 (+ 87us) server_negotiation.cc:407] Connection header received
0929 11:26:10.546396 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.546400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.546460 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.546562 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.547313 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.548032 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.549320 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.549755 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.549907 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.553116 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.553123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.553129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.553169 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.556912 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.557721 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.557727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.557730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.557889 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.558295 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.558300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.558303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.558809 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.558958 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.559396 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.559633 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:26:10.566541 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711: Bootstrap starting.
I20260929 11:26:10.570695 22859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:10.572041 22859 log.cc:824] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:10.575649 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711: No bootstrap required, opened a new log
I20260929 11:26:10.583577 22859 raft_consensus.cc:348] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.584026 22859 raft_consensus.cc:729] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b050af65c3fd4b1eb45acb2086f68711, State: Initialized, Role: FOLLOWER
I20260929 11:26:10.584553 22859 consensus_queue.cc:260] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.585608 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.585984 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.586946 22859 sys_catalog.cc:564] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:10.587007 22834 raft_consensus.cc:482] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 11:26:10.587209 22834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:10.590212 22834 raft_consensus.cc:504] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.592448 22982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b050af65c3fd4b1eb45acb2086f68711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0747aafb4f9400891dac3d06bd90128"
I20260929 11:26:10.593036 22982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:10.596602 22982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b050af65c3fd4b1eb45acb2086f68711 in term 1.
I20260929 11:26:10.597404 22810 leader_election.cc:302] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0747aafb4f9400891dac3d06bd90128, b050af65c3fd4b1eb45acb2086f68711; no voters:
I20260929 11:26:10.597878 23029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:10.597937 22834 leader_election.cc:288] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [CANDIDATE]: Term 1 election: Requested vote from peers 74e39a3c684643f29afaf14a05f8e79f (127.0.97.61:45111), a0747aafb4f9400891dac3d06bd90128 (127.0.97.60:44637)
I20260929 11:26:10.612958 23029 raft_consensus.cc:686] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [term 1 LEADER]: Becoming Leader. State: Replica: b050af65c3fd4b1eb45acb2086f68711, State: Running, Role: LEADER
I20260929 11:26:10.614518 23029 consensus_queue.cc:237] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.633241 23037 sys_catalog.cc:455] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b050af65c3fd4b1eb45acb2086f68711. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.633944 23037 sys_catalog.cc:458] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:10.640612 22933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } attempt: 5
W20260929 11:26:10.644451 22813 tablet.cc:2369] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:10.644764 23049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 11:26:10.647557 22961 tablet.cc:2369] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:10.663897 23049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:10.665913 23050 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:26:10.670641 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.630513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.0.97.61:45111)
0929 11:26:10.630702 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.630710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.643568 (+ 12858us) server_negotiation.cc:407] Connection header received
0929 11:26:10.643878 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.643885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.643967 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.644040 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.647456 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.648725 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.653384 (+ 4659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.654293 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.654416 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.661108 (+ 6692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.661115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.661119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.661167 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.664998 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.668107 (+ 3109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.668115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.668121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.668318 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.668830 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.668836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.668839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.669223 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.669453 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.670038 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.670267 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 11:26:10 dist-test-slave-srst krb5kdc[22779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 11:26:10.685559 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.624825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.97.62:39117)
0929 11:26:10.625027 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.625035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.629099 (+ 4064us) server_negotiation.cc:407] Connection header received
0929 11:26:10.630297 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.630306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.630431 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.630509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.658059 (+ 27550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.659148 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.666995 (+ 7847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.670995 (+ 4000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.671128 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.678144 (+ 7016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.678154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.678162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.678209 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.682224 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.683066 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.683072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.683075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.683243 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.683716 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.683722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.683725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.684083 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.684330 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.684912 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.685154 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:26:10.690135 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.628483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.97.60:44637)
0929 11:26:10.628678 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.628687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.630371 (+ 1684us) server_negotiation.cc:407] Connection header received
0929 11:26:10.631357 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.631364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.631443 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.631541 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.646702 (+ 15161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.647898 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.651985 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.652673 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.652783 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.661262 (+ 8479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.661268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.661274 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.661317 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.669527 (+ 8210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.671037 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.671074 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.671080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.671344 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.672005 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.672046 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.672050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.672782 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.675844 (+ 3062us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.675996 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.676254 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:26:10.676704 22792 master_runner.cc:186] Error getting master registration for 127.0.97.61:45111: OK, instance_id { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" instance_seqno: 1790681169911334 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:26:10.692620 22792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:26:10.698926 22982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [term 1 FOLLOWER]: Refusing update from remote peer b050af65c3fd4b1eb45acb2086f68711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:10.700080 23037 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:10.712646 23015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b050af65c3fd4b1eb45acb2086f68711. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.712978 23015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.718951 23053 mvcc.cc:205] Tried to move back new op lower bound from 7334630075172265984 to 7334630074854166528. Current Snapshot: MvccSnapshot[applied={T|T < 7334630075172265984}]
I20260929 11:26:10.719846 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.641211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44637 (local address 127.0.0.1:33324)
0929 11:26:10.641574 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:10.641592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:10.641704 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:10.668112 (+ 26408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:10.668118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:10.668615 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:10.668834 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.668846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.672243 (+ 3397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.672248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.673642 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:10.673656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.674753 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.674770 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:10.674985 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.675554 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:10.675574 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:10.679533 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:10.716360 (+ 36827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.716369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.716375 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.717095 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.717645 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:10.717653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:10.717659 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:10.717877 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:10.718616 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:10.718623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:10.718857 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 11:26:10.719014 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 11:26:10.719370 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260929 11:26:10.720745 23015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.720983 23015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.722357 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:10.641931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.97.60:44637)
0929 11:26:10.666583 (+ 24652us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.666588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.666616 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:26:10.666674 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.666679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.666758 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.668994 (+ 2236us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.669311 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.670649 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.673849 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:10.674583 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:10.675320 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:10.700248 (+ 24928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:10.700260 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:10.700268 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:10.700313 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:10.715915 (+ 15602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.717291 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.717300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.717306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.717499 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:10.718035 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:10.718043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:10.718049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:10.718459 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:10.721452 (+ 2993us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:10.721604 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 11:26:10.721887 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24459,"thread_start_us":24930,"threads_started":1}
I20260929 11:26:10.726076 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.726432 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:10.728143 22933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: Bootstrap starting.
I20260929 11:26:10.729645 23049 catalog_manager.cc:1347] Generated new cluster ID: 3deaf787120a499588925e1cdaba7d4e
I20260929 11:26:10.729759 23049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:10.732111 22933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:10.733439 22933 log.cc:824] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:10.767886 22933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: No bootstrap required, opened a new log
I20260929 11:26:10.785786 23049 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:10.787707 23049 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:10.793079 22933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.793884 22933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e39a3c684643f29afaf14a05f8e79f, State: Initialized, Role: FOLLOWER
I20260929 11:26:10.794821 22933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } }
I20260929 11:26:10.806734 23049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711: Generated new TSK 0
I20260929 11:26:10.807834 23049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:10.807483 23059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.808105 23059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.809887 22933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:10.810340 22907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:10.814181 22907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [term 1 FOLLOWER]: Refusing update from remote peer b050af65c3fd4b1eb45acb2086f68711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:10.814373 22908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b050af65c3fd4b1eb45acb2086f68711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74e39a3c684643f29afaf14a05f8e79f"
I20260929 11:26:10.818497 22908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b050af65c3fd4b1eb45acb2086f68711 in term 1.
I20260929 11:26:10.820111 23029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:10.859587 23059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b050af65c3fd4b1eb45acb2086f68711. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.859983 23059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.859776 23065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:10.860142 23065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:10.885721 23076 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 11:26:10.919109 23012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.909133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.0.97.62:39117)
0929 11:26:10.909714 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.909721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.916609 (+ 6888us) server_negotiation.cc:407] Connection header received
0929 11:26:10.916988 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.916997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.917141 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.917235 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.918751 (+ 1516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60708: BlockingRecv error: recv got EOF from 127.0.0.1:60708 (error 108)
Metrics: {"server-negotiator.queue_time_us":415}
W20260929 11:26:10.921715 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.909464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.97.61:45111)
0929 11:26:10.909887 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.909892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.919789 (+ 9897us) server_negotiation.cc:407] Connection header received
0929 11:26:10.920072 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.920081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.920177 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.920269 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.921387 (+ 1118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54782: BlockingRecv error: recv got EOF from 127.0.0.1:54782 (error 108)
Metrics: {"server-negotiator.queue_time_us":280}
W20260929 11:26:10.927591 23051 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.910107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.0.97.60:44637)
0929 11:26:10.910285 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.910291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.925622 (+ 15331us) server_negotiation.cc:407] Connection header received
0929 11:26:10.926182 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.926188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.926288 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.926356 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.927381 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33340: BlockingRecv error: recv got EOF from 127.0.0.1:33340 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 11:26:10.953919 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.951560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.97.61:45111)
0929 11:26:10.951765 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.951772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.952306 (+ 534us) server_negotiation.cc:407] Connection header received
0929 11:26:10.952551 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.952557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.952725 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.952838 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.953650 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54798: BlockingRecv error: recv got EOF from 127.0.0.1:54798 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 11:26:10.961138 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.959037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.97.61:45111)
0929 11:26:10.959208 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.959213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.959691 (+ 478us) server_negotiation.cc:407] Connection header received
0929 11:26:10.959895 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.959901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.959987 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.960091 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.960930 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54806: BlockingRecv error: recv got EOF from 127.0.0.1:54806 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 11:26:10.977072 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.974702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.97.61:45111)
0929 11:26:10.974906 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.974913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.975490 (+ 577us) server_negotiation.cc:407] Connection header received
0929 11:26:10.975714 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.975720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.975825 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.975949 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.976826 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54818: BlockingRecv error: recv got EOF from 127.0.0.1:54818 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 11:26:10.986265 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.984029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.0.97.61:45111)
0929 11:26:10.984252 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.984260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.984780 (+ 520us) server_negotiation.cc:407] Connection header received
0929 11:26:10.984988 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.984993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.985083 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.985191 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.986028 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54820: BlockingRecv error: recv got EOF from 127.0.0.1:54820 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 11:26:11.000000 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:10.997761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.0.97.61:45111)
0929 11:26:10.997949 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:10.997958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:10.998497 (+ 539us) server_negotiation.cc:407] Connection header received
0929 11:26:10.998710 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:10.998716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:10.998806 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:10.998912 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:10.999769 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54832: BlockingRecv error: recv got EOF from 127.0.0.1:54832 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 11:26:11.021899 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:11.019590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.0.97.61:45111)
0929 11:26:11.019797 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:11.019806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:11.020336 (+ 530us) server_negotiation.cc:407] Connection header received
0929 11:26:11.020559 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:11.020566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:11.020689 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:11.020786 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:11.021674 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54838: BlockingRecv error: recv got EOF from 127.0.0.1:54838 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 11:26:11.061676 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:11.059065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.0.97.61:45111)
0929 11:26:11.059289 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:11.059297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:11.059880 (+ 583us) server_negotiation.cc:407] Connection header received
0929 11:26:11.060100 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:11.060108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:11.060213 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:11.060332 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:11.061403 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54844: BlockingRecv error: recv got EOF from 127.0.0.1:54844 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 11:26:11.137938 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:11.135746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.0.97.61:45111)
0929 11:26:11.136011 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:11.136019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:11.136379 (+ 360us) server_negotiation.cc:407] Connection header received
0929 11:26:11.136638 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:11.136644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:11.136728 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:11.136817 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:11.137713 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54848: BlockingRecv error: recv got EOF from 127.0.0.1:54848 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 11:26:11.230213 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b050af65c3fd4b1eb45acb2086f68711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b050af65c3fd4b1eb45acb2086f68711" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39117 } } peers { permanent_uuid: "74e39a3c684643f29afaf14a05f8e79f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45111 } } peers { permanent_uuid: "a0747aafb4f9400891dac3d06bd90128" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44637 } } }
I20260929 11:26:11.230558 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b050af65c3fd4b1eb45acb2086f68711 [sys.catalog]: This master's current role is: LEADER
W20260929 11:26:11.271723 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:11.269807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.0.97.61:45111)
0929 11:26:11.270053 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:11.270060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:11.270358 (+ 298us) server_negotiation.cc:407] Connection header received
0929 11:26:11.270600 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:11.270604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:11.270676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:11.270790 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:11.271523 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54854: BlockingRecv error: recv got EOF from 127.0.0.1:54854 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:26:11.511829 23026 catalog_manager.cc:1259] Loaded cluster ID: 3deaf787120a499588925e1cdaba7d4e
I20260929 11:26:11.511945 23026 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: loading cluster ID for follower catalog manager: success
I20260929 11:26:11.514448 23026 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: acquiring CA information for follower catalog manager: success
I20260929 11:26:11.516275 23026 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0747aafb4f9400891dac3d06bd90128: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:26:11.533354 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:11.531488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.0.97.61:45111)
0929 11:26:11.531714 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:11.531721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:11.531918 (+ 197us) server_negotiation.cc:407] Connection header received
0929 11:26:11.532160 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:11.532164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:11.532234 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:11.532347 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:11.533138 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54866: BlockingRecv error: recv got EOF from 127.0.0.1:54866 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:26:11.890157 23076 catalog_manager.cc:1259] Loaded cluster ID: 3deaf787120a499588925e1cdaba7d4e
I20260929 11:26:11.890270 23076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: loading cluster ID for follower catalog manager: success
I20260929 11:26:11.892899 23076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: acquiring CA information for follower catalog manager: success
I20260929 11:26:11.894667 23076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74e39a3c684643f29afaf14a05f8e79f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:26:12.070147 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:12.062962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.0.97.61:45111)
0929 11:26:12.063837 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:12.063842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:12.063860 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:26:12.063922 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:12.063925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:12.064079 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:12.064197 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:12.064783 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.065735 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.067708 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.068171 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.068267 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:12.068897 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:12.069466 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:12.069592 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:12.069719 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 11:26:12.069788 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":572,"threads_started":1}
I20260929 11:26:12.082602 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:12.076398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.97.60:44637)
0929 11:26:12.077110 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:12.077114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:12.077131 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:26:12.077238 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:12.077242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:12.077446 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:12.077604 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:12.078013 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.078777 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.080251 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.080793 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.080939 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:12.081454 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:12.081973 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:12.082144 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:12.082195 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 11:26:12.082277 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":485,"threads_started":1}
I20260929 11:26:12.111620 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:12.090034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.97.62:39117)
0929 11:26:12.096992 (+ 6958us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:12.096998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:12.103758 (+ 6760us) server_negotiation.cc:407] Connection header received
0929 11:26:12.104101 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:12.104108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:12.104338 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:12.104490 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:12.105351 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.106533 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.108766 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.109387 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.109497 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:12.110324 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:12.110792 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:12.110887 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:12.111153 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 11:26:12.111251 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6801,"thread_start_us":6679,"threads_started":1}
I20260929 11:26:12.112409 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:12.089654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.0.97.61:45111)
0929 11:26:12.090338 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:12.090346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:12.105249 (+ 14903us) server_negotiation.cc:407] Connection header received
0929 11:26:12.105420 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:12.105429 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:12.105530 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:12.105634 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:12.106415 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.107589 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.110128 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.110816 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.110937 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:12.111587 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:12.111711 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:12.111772 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:12.111989 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 11:26:12.112070 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534}
I20260929 11:26:12.113016 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:12.089838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.0.97.60:44637)
0929 11:26:12.097104 (+ 7266us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:12.097110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:12.105897 (+ 8787us) server_negotiation.cc:407] Connection header received
0929 11:26:12.106061 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:12.106069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:12.106182 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:12.106282 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:12.107121 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.108266 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.110528 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:12.111232 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:12.111354 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:12.111954 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:12.112088 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:12.112170 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:12.112623 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 11:26:12.112711 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7140}
I20260929 11:26:12.119515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22792
I20260929 11:26:12.120644 22854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:12.233173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22792
W20260929 11:26:12.253829 22884 connection.cc:535] server connection from 127.0.0.1:59232 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:26:12.254546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22866
I20260929 11:26:12.255630 22928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:12.361750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22866
I20260929 11:26:12.382473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 22940
I20260929 11:26:12.383535 23002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:12.490624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 22940
2026-09-29T11:26:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4862 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:12 dist-test-slave-srst krb5kdc[23162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:12 dist-test-slave-srst krb5kdc[23162](info): set up 2 sockets
Sep 29 11:26:12 dist-test-slave-srst krb5kdc[23162](info): commencing operation
krb5kdc: starting...
W20260929 11:26:13.589190 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/test-user.keytab.
Sep 29 11:26:13 dist-test-slave-srst krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 11:26:13.848249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37251
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37251,127.0.97.61:41391,127.0.97.60:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:14.172158 23175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:26:14.172462 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:14.172525 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:14.172672 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:14.179816 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:14.179919 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:14.179977 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:14.180027 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:14.180083 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:14.190163 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37251,127.0.97.61:41391,127.0.97.60:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:14.191496 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:14.192783 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:14.203883 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:14.203986 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:14.203925 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:14.205264 23175 server_base.cc:1034] running on GCE node
I20260929 11:26:14.205901 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:14.207108 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:14.208320 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790681174208250 us; error 47 us; skew 500 ppm
I20260929 11:26:14.208688 23175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:14 dist-test-slave-srst krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:14.211551 23175 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 11:26:14.212981 23175 webserver.cc:460] Webserver started at http://127.0.97.62:36171/ using document root <none> and password file <none>
I20260929 11:26:14.213593 23175 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:14.213671 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:14.213961 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:14.216768 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data/instance:
uuid: "aa0930dbaa1d42369e1fd65d9e570f32"
format_stamp: "Formatted at 2026-09-29 11:26:14 on dist-test-slave-srst"
server_key: "09911b9da493001d84f0a304079dcf1b"
server_key_iv: "90be0e5ebc7be1a42b9f88b4b547aec9"
server_key_version: "encryptionkey@0"
I20260929 11:26:14.217473 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa0930dbaa1d42369e1fd65d9e570f32"
format_stamp: "Formatted at 2026-09-29 11:26:14 on dist-test-slave-srst"
server_key: "09911b9da493001d84f0a304079dcf1b"
server_key_iv: "90be0e5ebc7be1a42b9f88b4b547aec9"
server_key_version: "encryptionkey@0"
I20260929 11:26:14.222229 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:26:14.225350 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:14.226660 23175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:14.226842 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal
uuid: "aa0930dbaa1d42369e1fd65d9e570f32"
format_stamp: "Formatted at 2026-09-29 11:26:14 on dist-test-slave-srst"
server_key: "09911b9da493001d84f0a304079dcf1b"
server_key_iv: "90be0e5ebc7be1a42b9f88b4b547aec9"
server_key_version: "encryptionkey@0"
I20260929 11:26:14.227082 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:14.258515 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:14.262444 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:14.262722 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:14.281087 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37251
I20260929 11:26:14.281121 23241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37251 every 8 connection(s)
I20260929 11:26:14.282394 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:14.286103 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:14.292228 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 23175
I20260929 11:26:14.292446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-0/wal/instance
I20260929 11:26:14.292914 388 external_mini_cluster.cc:1428] Setting key 23bb31b78eb92a37aeda892e2db7e531
I20260929 11:26:14.292649 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:14 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:26:14.316919 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.295639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37251 (local address 127.0.0.1:33712)
0929 11:26:14.296561 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.296631 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:14.298580 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:14.299415 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:14.299423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:14.300124 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:14.300431 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.300447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.302609 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.302617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.304525 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.304548 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.305590 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.305599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.305805 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.306472 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:14.306504 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:14.309150 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:14.312389 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.312399 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.312412 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.312878 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.313276 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.313282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.313286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.313455 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.314312 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:14.314320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:14.314554 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 11:26:14.314799 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 11:26:14.315492 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":164,"thread_start_us":479,"threads_started":1}
I20260929 11:26:14.316951 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.295640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.97.62:37251)
0929 11:26:14.297398 (+ 1758us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:14.297411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:14.297469 (+ 58us) server_negotiation.cc:407] Connection header received
0929 11:26:14.298948 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:14.298955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:14.299208 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:14.299408 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:14.300666 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.302452 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.304785 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.305335 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.305473 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.309384 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:14.309392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:14.309405 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:14.309433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:14.312236 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.313007 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.313016 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.313026 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.313166 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.313572 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.313583 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.313585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.314155 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:14.314397 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:14.314937 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 11:26:14.315365 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":1549,"thread_start_us":1334,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260929 11:26:14.319912 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 11:26:14.321197 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.320718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57088)
0929 11:26:14.320892 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.321023 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 11:26:14.321506 23191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
I20260929 11:26:14.323171 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.322713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57092)
0929 11:26:14.322904 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.322998 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:26:14.323374 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41391: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 11:26:14.352059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41391
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37251,127.0.97.61:41391,127.0.97.60:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:26:14.372860 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } attempt: 1
I20260929 11:26:14.374153 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.373584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57100)
0929 11:26:14.373831 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.373951 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 11:26:14.375279 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.374816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57102)
0929 11:26:14.374983 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.375095 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:26:14.375506 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41391: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
I20260929 11:26:14.443789 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } attempt: 2
I20260929 11:26:14.445035 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.444534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57116)
0929 11:26:14.444799 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.444896 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 11:26:14.446076 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.445753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57130)
0929 11:26:14.445876 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.445952 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:26:14.446336 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41391: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
I20260929 11:26:14.537575 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } attempt: 3
I20260929 11:26:14.538724 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.538320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57138)
0929 11:26:14.538507 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.538595 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:26:14.539983 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.539498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57146)
0929 11:26:14.539682 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.539777 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 11:26:14.540261 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41391: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
I20260929 11:26:14.683481 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } attempt: 4
I20260929 11:26:14.684815 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.684168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57152)
0929 11:26:14.684532 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.684681 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 11:26:14.685765 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.685442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57154)
0929 11:26:14.685579 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.685660 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:26:14.685945 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41391: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41391: connect: Connection refused (error 111)
W20260929 11:26:14.690358 23249 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:26:14.690692 23249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:14.690814 23249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:14.690948 23249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:14.698092 23249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:14.698210 23249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:14.698321 23249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:14.698403 23249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:14.698508 23249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:14.708139 23249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37251,127.0.97.61:41391,127.0.97.60:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:14.709506 23249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:14.710798 23249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:14.722513 23254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:14.722481 23257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:14.722422 23255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:14.723281 23249 server_base.cc:1034] running on GCE node
I20260929 11:26:14.723989 23249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:14.725521 23249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:14.726733 23249 hybrid_clock.cc:648] HybridClock initialized: now 1790681174726651 us; error 53 us; skew 500 ppm
I20260929 11:26:14.727058 23249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:14 dist-test-slave-srst krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:14.730268 23249 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 11:26:14.731696 23249 webserver.cc:460] Webserver started at http://127.0.97.61:33791/ using document root <none> and password file <none>
I20260929 11:26:14.732389 23249 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:14.732476 23249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:14.732795 23249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:14.735692 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data/instance:
uuid: "9da3c48500ec4c97a4af8fee6c29c71d"
format_stamp: "Formatted at 2026-09-29 11:26:14 on dist-test-slave-srst"
server_key: "ecf76f10ab36886ae05f6c058514c95b"
server_key_iv: "a3f4ff82b26a3e56dc0f66ff9d7d54b2"
server_key_version: "encryptionkey@0"
I20260929 11:26:14.736403 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal/instance:
uuid: "9da3c48500ec4c97a4af8fee6c29c71d"
format_stamp: "Formatted at 2026-09-29 11:26:14 on dist-test-slave-srst"
server_key: "ecf76f10ab36886ae05f6c058514c95b"
server_key_iv: "a3f4ff82b26a3e56dc0f66ff9d7d54b2"
server_key_version: "encryptionkey@0"
I20260929 11:26:14.741309 23249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:26:14.744511 23264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:14.745961 23249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:26:14.746122 23249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal
uuid: "9da3c48500ec4c97a4af8fee6c29c71d"
format_stamp: "Formatted at 2026-09-29 11:26:14 on dist-test-slave-srst"
server_key: "ecf76f10ab36886ae05f6c058514c95b"
server_key_iv: "a3f4ff82b26a3e56dc0f66ff9d7d54b2"
server_key_version: "encryptionkey@0"
I20260929 11:26:14.746317 23249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:14.760896 23249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:14.764756 23249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:14.765031 23249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:14.782929 23249 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41391
I20260929 11:26:14.782948 23315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41391 every 8 connection(s)
I20260929 11:26:14.784229 23249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/data/info.pb
I20260929 11:26:14.786446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 23249
I20260929 11:26:14.786638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-1/wal/instance
I20260929 11:26:14.787199 388 external_mini_cluster.cc:1428] Setting key c6dd453a811ca240ca75462faf3ee371
I20260929 11:26:14.788671 23316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:14.798458 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:14 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 11:26:14.819295 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.801236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.0.97.62:37251)
0929 11:26:14.801471 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:14.801479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:14.802660 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 11:26:14.804454 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:14.804463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:14.804558 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:14.804712 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:14.806117 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.807271 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.808752 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.809535 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.809684 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.813075 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:14.813084 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:14.813091 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:14.813129 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:14.816080 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.817052 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.817056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.817058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.817174 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.817636 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.817639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.817641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.817908 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:14.818022 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:14.818898 (+ 876us) server_negotiation.cc:299] Negotiation successful
0929 11:26:14.819061 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:26:14.820783 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.801615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37251 (local address 127.0.0.1:33728)
0929 11:26:14.802483 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.802533 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:14.804191 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:14.804898 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:14.804906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:14.805418 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:14.805948 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.805964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.807417 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.807421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.808562 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.808573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.809741 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.809746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.809953 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.810663 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:14.810689 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:14.812864 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:14.816233 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.816241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.816255 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.816945 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.817283 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.817291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.817298 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.817530 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.818036 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:14.818046 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:14.818514 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 11:26:14.818816 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 11:26:14.819358 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":426,"threads_started":1}
I20260929 11:26:14.822538 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:14 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:26:14.838896 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.823326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57158)
0929 11:26:14.823534 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.823553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:14.823659 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:14.824686 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:14.824694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:14.825161 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:14.825301 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.825309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.827451 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.827455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.828278 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.828288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.829525 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.829529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.829661 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.830286 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:14.830303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:14.832831 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:14.835918 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.835926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.835933 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.836483 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.836905 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.836913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.836919 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.837094 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.837663 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:14.837671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:14.837849 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 11:26:14.838012 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 11:26:14.838365 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:26:14.839052 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.823512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.0.97.61:41391)
0929 11:26:14.824211 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:14.824217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:14.824240 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:26:14.824280 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:14.824285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:14.824463 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:14.824577 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:14.825476 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.827283 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.828421 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.829395 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.829537 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.833032 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:14.833043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:14.833056 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:14.833084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:14.835784 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.836660 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.836667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.836669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.836787 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.837199 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.837202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.837204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.837529 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:14.837678 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:14.838133 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 11:26:14.838536 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":551,"thread_start_us":407,"threads_started":1}
I20260929 11:26:14.842613 23316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:26:14.843865 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.843427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43114)
0929 11:26:14.843599 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.843717 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:26:14.844126 23266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:14.845772 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.845375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43128)
0929 11:26:14.845547 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.845655 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:26:14.845986 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 11:26:14.850068 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
/tmp/dist-test-taskyNUK1V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34719
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37251,127.0.97.61:41391,127.0.97.60:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:26:14.895326 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 1
I20260929 11:26:14.896576 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.896012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43142)
0929 11:26:14.896251 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.896384 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 11:26:14.897624 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.897215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43144)
0929 11:26:14.897388 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.897498 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 11:26:14.897850 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:14.966096 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 2
I20260929 11:26:14.967455 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.966796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43146)
0929 11:26:14.967060 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.967206 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 11:26:14.968758 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.968147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43156)
0929 11:26:14.968438 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.968553 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 11:26:14.969189 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:14.985177 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } attempt: 5
Sep 29 11:26:14 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:26:14.999217 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.985936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.0.97.61:41391)
0929 11:26:14.986133 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:14.986137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:14.986156 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:26:14.986375 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:14.986379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:14.986440 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:14.986512 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:14.987589 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.988245 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.990198 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.990613 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.990705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.994090 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:14.994094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:14.994098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:14.994127 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:14.996810 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.997417 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.997424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.997427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.997548 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:14.997901 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:14.997907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:14.997910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:14.998174 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:14.998319 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:14.998693 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 11:26:14.998902 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:26:14.999238 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:14.985834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57170)
0929 11:26:14.986075 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:14.986098 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:14.986235 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:14.986587 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:14.986592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:14.987115 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:14.987431 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.987446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.988407 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.988413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.990022 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:14.990042 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:14.990776 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:14.990782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:14.990999 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:14.991631 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:14.991653 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:14.993906 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:14.996939 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.996944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.996948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.997302 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.997656 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:14.997661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:14.997665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:14.997798 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:14.998298 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:14.998303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:14.998452 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 11:26:14.998590 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 11:26:14.998847 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 11:26:15.000458 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:26:15.001463 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.001120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43162)
0929 11:26:15.001268 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.001336 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:26:15.002378 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.002069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43168)
0929 11:26:15.002204 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.002288 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:26:15.002585 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:15.053779 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 1
I20260929 11:26:15.055115 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.054412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43182)
0929 11:26:15.054732 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.054869 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 11:26:15.056306 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.055828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43194)
0929 11:26:15.056018 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.056123 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 11:26:15.056530 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:15.060428 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 3
I20260929 11:26:15.061589 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.061003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43198)
0929 11:26:15.061247 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.061376 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:26:15.062750 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.062267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43200)
0929 11:26:15.062508 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.062605 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260929 11:26:15.062974 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:15.124778 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 2
I20260929 11:26:15.126078 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.125469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43214)
0929 11:26:15.125729 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.125850 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:26:15.127170 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.126751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43216)
0929 11:26:15.126920 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.127018 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 11:26:15.127400 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
W20260929 11:26:15.179651 23323 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:26:15.179975 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:15.180054 23323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:15.180145 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:15.187481 23323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:15.187578 23323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:15.187641 23323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:15.187700 23323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:15.187767 23323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:15.197475 23323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37251,127.0.97.61:41391,127.0.97.60:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:15.198840 23323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:15.200222 23323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:26:15.206200 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 4
I20260929 11:26:15.207412 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.206932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43226)
0929 11:26:15.207182 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.207271 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 11:26:15.208381 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.208058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43230)
0929 11:26:15.208200 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.208282 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 11:26:15.208621 23316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
W20260929 11:26:15.212766 23331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:15.212890 23328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:15.213056 23329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:15.214342 23323 server_base.cc:1034] running on GCE node
I20260929 11:26:15.215127 23323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:15.216477 23323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:15.217689 23323 hybrid_clock.cc:648] HybridClock initialized: now 1790681175217608 us; error 53 us; skew 500 ppm
I20260929 11:26:15.218032 23323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:15.221019 23323 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 11:26:15.222340 23323 webserver.cc:460] Webserver started at http://127.0.97.60:41873/ using document root <none> and password file <none>
I20260929 11:26:15.223034 23323 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:15.223160 23323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:15.223487 23323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:15.226401 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data/instance:
uuid: "a2623d4bb5ae422b85c7782d06c521f5"
format_stamp: "Formatted at 2026-09-29 11:26:15 on dist-test-slave-srst"
server_key: "a846988839bdbd123a16604ede199006"
server_key_iv: "dbd402aba85b53299796c2eb20e09291"
server_key_version: "encryptionkey@0"
I20260929 11:26:15.227180 23323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal/instance:
uuid: "a2623d4bb5ae422b85c7782d06c521f5"
format_stamp: "Formatted at 2026-09-29 11:26:15 on dist-test-slave-srst"
server_key: "a846988839bdbd123a16604ede199006"
server_key_iv: "dbd402aba85b53299796c2eb20e09291"
server_key_version: "encryptionkey@0"
I20260929 11:26:15.232357 23323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:26:15.233613 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 3
I20260929 11:26:15.234576 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.234130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43236)
0929 11:26:15.234354 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.234429 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:26:15.235664 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.235249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43248)
0929 11:26:15.235434 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.235497 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 11:26:15.235939 23242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34719: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34719: connect: Connection refused (error 111)
I20260929 11:26:15.236047 23338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:15.237490 23323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:15.237651 23323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal
uuid: "a2623d4bb5ae422b85c7782d06c521f5"
format_stamp: "Formatted at 2026-09-29 11:26:15 on dist-test-slave-srst"
server_key: "a846988839bdbd123a16604ede199006"
server_key_iv: "dbd402aba85b53299796c2eb20e09291"
server_key_version: "encryptionkey@0"
I20260929 11:26:15.237864 23323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:15.262622 23323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:15.266664 23323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:15.266957 23323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:15.285482 23389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34719 every 8 connection(s)
I20260929 11:26:15.285512 23323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34719
I20260929 11:26:15.287209 23323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/data/info.pb
I20260929 11:26:15.290891 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:15.294476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu as pid 23323
I20260929 11:26:15.294672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyNUK1V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790680821855275-388-0/minicluster-data/master-2/wal/instance
I20260929 11:26:15.295105 388 external_mini_cluster.cc:1428] Setting key 826cb2a213979738103c4a64f433ba2c
I20260929 11:26:15.297994 23390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:26:15.317727 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.301819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.0.97.62:37251)
0929 11:26:15.301987 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.301991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.302712 (+ 721us) server_negotiation.cc:407] Connection header received
0929 11:26:15.302898 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.302902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.302968 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.303048 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.304593 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.305758 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.307694 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.308301 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.308440 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.312266 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.312271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.312274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.312307 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.314809 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.315507 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.315512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.315516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.315658 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.316043 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.316050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.316056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.316352 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.316476 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.317126 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.317349 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:26:15.319211 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.296894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.97.60:34719)
0929 11:26:15.297691 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.297706 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.297754 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:26:15.299861 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.299888 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.300375 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.300791 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.302465 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.304459 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.306178 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.307303 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.307508 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.311694 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.311710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.311727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.311765 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.314930 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.315564 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.315570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.315572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.315728 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.316099 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.316102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.316104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.316720 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.317047 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.317639 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.317863 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":341,"threads_started":1}
I20260929 11:26:15.319248 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.301902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37251 (local address 127.0.0.1:33740)
0929 11:26:15.302556 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.302597 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:15.302770 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:15.303150 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:15.303183 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:15.303982 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:15.304359 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.304374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.305903 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.305907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.307553 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.307565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.308504 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.308509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.308731 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.309351 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:15.309377 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:15.312070 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:15.314945 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.314955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.314970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.315391 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.315763 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.315770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.315775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.315936 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.316487 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:15.316494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:15.316736 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 11:26:15.317009 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 11:26:15.317493 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":298,"threads_started":1}
I20260929 11:26:15.320811 23390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 11:26:15.334230 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.321614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41391 (local address 127.0.0.1:57184)
0929 11:26:15.321921 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.321941 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:15.322065 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:15.322451 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:15.322457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:15.322926 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:15.323126 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.323140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.324104 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.324111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.325125 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.325159 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.325808 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.325814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.326004 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.326545 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:15.326571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:15.328542 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:15.331867 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.331872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.331876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.332219 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.332646 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.332651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.332655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.332796 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.333309 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:15.333315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:15.333463 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 11:26:15.333596 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 11:26:15.333872 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260929 11:26:15.334271 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.321609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.0.97.61:41391)
0929 11:26:15.321822 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.321826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.322046 (+ 220us) server_negotiation.cc:407] Connection header received
0929 11:26:15.322194 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.322198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.322270 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.322417 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.323285 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.323958 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.325281 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.325674 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.325815 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.328757 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.328762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.328766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.328797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.331732 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.332343 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.332356 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.332359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.332493 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.332918 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.332928 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.332930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.333192 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.333350 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.333711 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.333914 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:26:15.335274 23390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:26:15.337018 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.322672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.0.97.62:37251)
0929 11:26:15.322818 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.322822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.322837 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:26:15.323005 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.323009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.323062 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.323134 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.324316 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.325085 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.326585 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.327053 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.327144 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.331550 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.331556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.331560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.331621 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.334257 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.335019 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.335026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.335029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.335189 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.335575 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.335581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.335584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.335901 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.336212 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.336491 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.336737 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 11:26:15.348351 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.336047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.0.97.60:34719)
0929 11:26:15.336220 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.336226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.336361 (+ 135us) server_negotiation.cc:407] Connection header received
0929 11:26:15.336513 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.336518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.336582 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.336705 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.337476 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.338303 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.339424 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.339831 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.339958 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.343224 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.343230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.343235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.343270 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.345891 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.346678 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.346681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.346684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.346781 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.347154 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.347157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.347159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.347429 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.347553 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.347968 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.348119 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":75}
I20260929 11:26:15.348768 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.336035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43278)
0929 11:26:15.336284 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.336304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:15.336406 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:15.336733 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:15.336738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:15.337188 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:15.337348 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.337356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.338435 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.338440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.339292 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.339302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.339965 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.339970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.340237 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.340785 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:15.340801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:15.343061 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:15.346030 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.346037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.346042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.346554 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.346874 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.346881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.346884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.347039 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.347562 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:15.347568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:15.347722 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 11:26:15.347905 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 11:26:15.348283 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":76}
I20260929 11:26:15.354105 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.338560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.0.97.61:41391)
0929 11:26:15.338708 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.338712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.338904 (+ 192us) server_negotiation.cc:407] Connection header received
0929 11:26:15.339069 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.339074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.339142 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.339244 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.340344 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.341359 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.343025 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.343648 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.343783 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.348486 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.348493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.348499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.348539 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.351748 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.352327 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.352332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.352335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.352484 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.352867 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.352872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.352876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.353188 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.353460 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.353611 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.353816 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:26:15.355021 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: Bootstrap starting.
I20260929 11:26:15.358791 23390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:15.358834 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:15.360046 23390 log.cc:824] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:15.362671 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: No bootstrap required, opened a new log
I20260929 11:26:15.361733 23217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aa0930dbaa1d42369e1fd65d9e570f32"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33756
I20260929 11:26:15.369581 23390 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.369892 23390 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2623d4bb5ae422b85c7782d06c521f5, State: Initialized, Role: FOLLOWER
I20260929 11:26:15.370350 23390 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.371228 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.371606 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.372103 23390 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:26:15.380360 23407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:26:15.380453 23407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:26:15.378957 23323 master_runner.cc:186] Error getting master registration for 127.0.97.62:37251: OK, instance_id { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" instance_seqno: 1790681174275966 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:26:15.389927 23323 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:26:15.413182 23242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 4
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 11:26:15.430050 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.413935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43288)
0929 11:26:15.414160 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.414183 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:15.414320 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:15.414775 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:15.414782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:15.415331 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:15.415564 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.415576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.417014 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.417020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.418507 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.418523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.420383 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.420390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.420626 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.421350 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:15.421375 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:15.424251 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:15.427506 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.427513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.427517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.427872 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.428264 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.428269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.428273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.428426 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.429058 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:15.429064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:15.429237 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 11:26:15.429378 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 11:26:15.429668 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 11:26:15.430056 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.413989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.0.97.60:34719)
0929 11:26:15.414252 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.414261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.414287 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:26:15.414479 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.414486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.414574 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.414681 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.415722 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.416861 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.418664 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.420237 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.420360 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.424448 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.424452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.424455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.424484 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.427375 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.428029 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.428041 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.428043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.428154 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.428535 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.428542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.428545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.428930 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.429108 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.429485 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.429696 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:26:15.436297 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32: Bootstrap starting.
I20260929 11:26:15.440826 23242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:15.442245 23242 log.cc:824] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:15.445753 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32: No bootstrap required, opened a new log
I20260929 11:26:15.453647 23242 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.454087 23242 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0930dbaa1d42369e1fd65d9e570f32, State: Initialized, Role: FOLLOWER
I20260929 11:26:15.454602 23242 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.455651 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.456032 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.456916 23242 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:15.456975 23217 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 11:26:15.457209 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:15.459965 23217 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.461419 23217 leader_election.cc:288] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [CANDIDATE]: Term 1 election: Requested vote from peers 9da3c48500ec4c97a4af8fee6c29c71d (127.0.97.61:41391), a2623d4bb5ae422b85c7782d06c521f5 (127.0.97.60:34719)
I20260929 11:26:15.462323 23365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a2623d4bb5ae422b85c7782d06c521f5"
I20260929 11:26:15.462826 23365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:15.466364 23365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0930dbaa1d42369e1fd65d9e570f32 in term 1.
I20260929 11:26:15.466992 23192 leader_election.cc:302] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2623d4bb5ae422b85c7782d06c521f5, aa0930dbaa1d42369e1fd65d9e570f32; no voters:
I20260929 11:26:15.467422 23410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:15.472887 23410 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [term 1 LEADER]: Becoming Leader. State: Replica: aa0930dbaa1d42369e1fd65d9e570f32, State: Running, Role: LEADER
I20260929 11:26:15.473400 23410 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.486503 23421 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0930dbaa1d42369e1fd65d9e570f32. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.486976 23421 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:15.488198 23425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:15.507742 23425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:15.507808 23316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } attempt: 5
W20260929 11:26:15.515585 23196 tablet.cc:2369] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:26:15.519019 23344 tablet.cc:2369] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:15.519862 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.466307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.97.61:41391)
0929 11:26:15.467282 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.467290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.481469 (+ 14179us) server_negotiation.cc:407] Connection header received
0929 11:26:15.483030 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.483040 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.483163 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.483251 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.490166 (+ 6915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.491243 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.497514 (+ 6271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.498406 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.498523 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.506320 (+ 7797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.506329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.506333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.506378 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.510013 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.510872 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.510878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.510881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.511035 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.511520 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.511524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.511527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.511898 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.512137 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.518257 (+ 6120us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.518502 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754}
I20260929 11:26:15.523105 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.465897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.0.97.62:37251)
0929 11:26:15.466076 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.466084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.487410 (+ 21326us) server_negotiation.cc:407] Connection header received
0929 11:26:15.487763 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.487770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.487853 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.487926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.493773 (+ 5847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.495121 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.501857 (+ 6736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.502637 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.502754 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.512537 (+ 9783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.512545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.512552 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.512627 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.516932 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.520675 (+ 3743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.520680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.520683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.520869 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.521320 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.521325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.521327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.521668 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.521944 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.522467 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.522724 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:26:15.525856 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.467764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.0.97.60:34719)
0929 11:26:15.467976 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.467988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.486233 (+ 18245us) server_negotiation.cc:407] Connection header received
0929 11:26:15.486414 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.486421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.486507 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.486615 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.489997 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.491165 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.494912 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.495624 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.495750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.502323 (+ 6573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.502330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.502336 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.502376 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.506234 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.507514 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.507519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.507523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.507674 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.524060 (+ 16386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.524067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.524073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.524408 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.524675 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.525193 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.525452 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 11:26:15 dist-test-slave-srst krb5kdc[23162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 11:26:15.534570 23365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [term 1 FOLLOWER]: Refusing update from remote peer aa0930dbaa1d42369e1fd65d9e570f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:15.535537 23421 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:15.544275 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.508357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34719 (local address 127.0.0.1:43298)
0929 11:26:15.509098 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:15.509124 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:15.509229 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:15.521520 (+ 12291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:15.521526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:15.522059 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:15.522257 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.522268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.526909 (+ 4641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.526915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.528357 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:15.528372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.529613 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.529620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:15.529841 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.530495 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:15.530520 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:15.533376 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:15.539861 (+ 6485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.539867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.539871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.540389 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.542200 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:15.542206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:15.542210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:15.542383 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:15.543200 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:15.543205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:15.543390 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 11:26:15.543565 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 11:26:15.543883 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260929 11:26:15.544863 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.508787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.0.97.60:34719)
0929 11:26:15.520827 (+ 12040us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.520832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.520854 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:26:15.520902 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.520906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.520975 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.522399 (+ 1424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.522660 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.523903 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.528543 (+ 4640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.529266 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.530177 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.533706 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:15.534231 (+ 525us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:15.534241 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:15.534381 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:15.539707 (+ 5326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.541829 (+ 2122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.541838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.541844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.542052 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:15.542637 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:15.542642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:15.542646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:15.543023 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:15.543181 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.543952 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.544230 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11897,"thread_start_us":12262,"threads_started":1}
I20260929 11:26:15.553359 23316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: Bootstrap starting.
I20260929 11:26:15.560278 23316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:15.562942 23316 log.cc:824] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:15.561388 23175 master_runner.cc:186] Error getting master registration for 127.0.97.61:41391: OK, instance_id { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" instance_seqno: 1790681174777920 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:26:15.572171 23175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:26:15.574321 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0930dbaa1d42369e1fd65d9e570f32. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.574610 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.577323 23430 mvcc.cc:205] Tried to move back new op lower bound from 7334630094967193600 to 7334630094744776704. Current Snapshot: MvccSnapshot[applied={T|T < 7334630094967193600}]
I20260929 11:26:15.580425 23316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: No bootstrap required, opened a new log
I20260929 11:26:15.596855 23316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.596925 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.597287 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:15.597348 23316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da3c48500ec4c97a4af8fee6c29c71d, State: Initialized, Role: FOLLOWER
I20260929 11:26:15.597862 23316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } }
I20260929 11:26:15.602877 23425 catalog_manager.cc:1347] Generated new cluster ID: b3fe7247036e41df8e75fcdc6d4e3a57
I20260929 11:26:15.602978 23425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:15.604797 23440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.605168 23440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.606884 23316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:15.606977 23291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9da3c48500ec4c97a4af8fee6c29c71d"
I20260929 11:26:15.607422 23291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:15.610872 23291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0930dbaa1d42369e1fd65d9e570f32 in term 1.
I20260929 11:26:15.611773 23290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [term 1 FOLLOWER]: Refusing update from remote peer aa0930dbaa1d42369e1fd65d9e570f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:15.613775 23421 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:15.621845 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.622116 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.644428 23425 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:15.646437 23425 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:15.673787 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.674139 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:15.677290 23440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0930dbaa1d42369e1fd65d9e570f32. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.677664 23440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.677265 23450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0930dbaa1d42369e1fd65d9e570f32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } peers { permanent_uuid: "9da3c48500ec4c97a4af8fee6c29c71d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41391 } } peers { permanent_uuid: "a2623d4bb5ae422b85c7782d06c521f5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34719 } } }
I20260929 11:26:15.678421 23450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:26:15.701824 23425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0930dbaa1d42369e1fd65d9e570f32: Generated new TSK 0
I20260929 11:26:15.702929 23425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:15.722774 23455 catalog_manager.cc:1259] Loaded cluster ID: b3fe7247036e41df8e75fcdc6d4e3a57
I20260929 11:26:15.722913 23455 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: loading cluster ID for follower catalog manager: success
I20260929 11:26:15.738116 23455 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: acquiring CA information for follower catalog manager: success
I20260929 11:26:15.740980 23455 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9da3c48500ec4c97a4af8fee6c29c71d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:26:15.842471 23320 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:15.819983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.0.97.61:41391)
0929 11:26:15.820219 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.820229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.829246 (+ 9017us) server_negotiation.cc:407] Connection header received
0929 11:26:15.830358 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.830368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.830503 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.830593 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.842180 (+ 11587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57206: BlockingRecv error: recv got EOF from 127.0.0.1:57206 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 11:26:15.844125 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:15.824765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.97.60:34719)
0929 11:26:15.824971 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.824977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.841512 (+ 16535us) server_negotiation.cc:407] Connection header received
0929 11:26:15.841911 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.841917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.842043 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.842128 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.843900 (+ 1772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43310: BlockingRecv error: recv got EOF from 127.0.0.1:43310 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 11:26:15.844247 23245 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:15.830653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.0.97.62:37251)
0929 11:26:15.830860 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.830866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.842723 (+ 11857us) server_negotiation.cc:407] Connection header received
0929 11:26:15.842822 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.842831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.842926 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.843030 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.843999 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33774: BlockingRecv error: recv got EOF from 127.0.0.1:33774 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:26:15.901880 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.882692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.0.97.61:41391)
0929 11:26:15.882886 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.882893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.893654 (+ 10761us) server_negotiation.cc:407] Connection header received
0929 11:26:15.894248 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.894255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.894514 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.894687 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:15.895422 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.896568 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.898655 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.899354 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:15.899466 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:15.900230 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:15.901007 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:15.901130 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:15.901377 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 11:26:15.901497 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 11:26:15.948544 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:15.935240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.0.97.60:34719)
0929 11:26:15.935480 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.935488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.946240 (+ 10752us) server_negotiation.cc:407] Connection header received
0929 11:26:15.946845 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.946851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.946992 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.947092 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:15.948250 (+ 1158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43326: BlockingRecv error: recv got EOF from 127.0.0.1:43326 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:26:16.002631 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:15.984455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.0.97.61:41391)
0929 11:26:15.984679 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:15.984686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:15.995392 (+ 10706us) server_negotiation.cc:407] Connection header received
0929 11:26:15.995908 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:15.995914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:15.996005 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:15.996081 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:15.996754 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:15.997942 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.000110 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.000879 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.000979 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.001652 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.001826 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.001892 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.002119 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.002227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 11:26:16.048767 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.036140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.97.60:34719)
0929 11:26:16.036311 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.036317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.047157 (+ 10840us) server_negotiation.cc:407] Connection header received
0929 11:26:16.047487 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.047493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.047590 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.047717 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.048521 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43330: BlockingRecv error: recv got EOF from 127.0.0.1:43330 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:26:16.059685 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.053761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.0.97.61:41391)
0929 11:26:16.053997 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.054011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.054512 (+ 501us) server_negotiation.cc:407] Connection header received
0929 11:26:16.054732 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.054738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.054825 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.054946 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.055351 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.056155 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.057564 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.058221 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.058369 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.058876 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.059001 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.059125 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.059267 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.059343 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 11:26:16.067504 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.065889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.0.97.60:34719)
0929 11:26:16.066058 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.066067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.066150 (+ 83us) server_negotiation.cc:407] Connection header received
0929 11:26:16.066420 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.066425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.066509 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.066620 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.067322 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43332: BlockingRecv error: recv got EOF from 127.0.0.1:43332 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:26:16.084059 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.078269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.0.97.61:41391)
0929 11:26:16.078578 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.078584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.079122 (+ 538us) server_negotiation.cc:407] Connection header received
0929 11:26:16.079363 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.079369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.079448 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.079550 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.079989 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.080825 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.082183 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.082688 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.082839 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.083252 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.083360 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.083460 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.083618 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.083704 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
W20260929 11:26:16.092100 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.090315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.0.97.60:34719)
0929 11:26:16.090540 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.090546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.090567 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:26:16.090771 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.090776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.090875 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.090936 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.091816 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43340: BlockingRecv error: recv got EOF from 127.0.0.1:43340 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 11:26:16.108858 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.103074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.0.97.61:41391)
0929 11:26:16.103307 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.103317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.103812 (+ 495us) server_negotiation.cc:407] Connection header received
0929 11:26:16.104040 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.104044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.104122 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.104232 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.104637 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.105419 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.106749 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.107264 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.107441 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.107878 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.108066 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.108176 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.108406 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.108488 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 11:26:16.118498 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.115495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.0.97.60:34719)
0929 11:26:16.115737 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.115746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.117130 (+ 1384us) server_negotiation.cc:407] Connection header received
0929 11:26:16.117336 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.117345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.117422 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.117529 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.118254 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 11:26:16.144801 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.138553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.0.97.61:41391)
0929 11:26:16.138819 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.138828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.139419 (+ 591us) server_negotiation.cc:407] Connection header received
0929 11:26:16.139662 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.139666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.139736 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.139849 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.140275 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.141386 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.142951 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.143432 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.143588 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.144013 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.144131 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.144236 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.144386 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.144477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 11:26:16.160881 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.158656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.97.60:34719)
0929 11:26:16.158882 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.158890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.159426 (+ 536us) server_negotiation.cc:407] Connection header received
0929 11:26:16.159650 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.159656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.159743 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.159857 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.160634 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43350: BlockingRecv error: recv got EOF from 127.0.0.1:43350 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:26:16.205080 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.198583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.0.97.61:41391)
0929 11:26:16.198858 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.198865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.199228 (+ 363us) server_negotiation.cc:407] Connection header received
0929 11:26:16.199460 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.199464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.199551 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.199677 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.200201 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.201133 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.203250 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.203652 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.203750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.204280 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.204381 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.204480 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.204679 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.204771 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 11:26:16.213680 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.211627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.0.97.60:34719)
0929 11:26:16.211839 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.211848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.212199 (+ 351us) server_negotiation.cc:407] Connection header received
0929 11:26:16.212412 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.212422 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.212496 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.212631 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.213411 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43352: BlockingRecv error: recv got EOF from 127.0.0.1:43352 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:26:16.286492 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.281032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.0.97.61:41391)
0929 11:26:16.281231 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.281239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.281600 (+ 361us) server_negotiation.cc:407] Connection header received
0929 11:26:16.281822 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.281827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.281913 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.282012 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.282458 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.283184 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.284523 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.285176 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.285305 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.285758 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.285891 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.285981 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.286133 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.286210 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 11:26:16.296407 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:16.294312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.97.60:34719)
0929 11:26:16.294488 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.294493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.295028 (+ 535us) server_negotiation.cc:407] Connection header received
0929 11:26:16.295179 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.295184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.295275 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.295395 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:16.296206 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43362: BlockingRecv error: recv got EOF from 127.0.0.1:43362 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:26:16.382471 23407 catalog_manager.cc:1259] Loaded cluster ID: b3fe7247036e41df8e75fcdc6d4e3a57
I20260929 11:26:16.382578 23407 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: loading cluster ID for follower catalog manager: success
I20260929 11:26:16.385252 23407 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: acquiring CA information for follower catalog manager: success
I20260929 11:26:16.387010 23407 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2623d4bb5ae422b85c7782d06c521f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:26:16.432801 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.427613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.0.97.61:41391)
0929 11:26:16.427875 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.427883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.428164 (+ 281us) server_negotiation.cc:407] Connection header received
0929 11:26:16.428404 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.428410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.428481 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.428582 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.429034 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.429757 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.431121 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.431528 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.431658 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.432051 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.432153 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.432251 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.432394 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.432473 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 11:26:16.444218 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.438132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.97.60:34719)
0929 11:26:16.438332 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.438339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.438656 (+ 317us) server_negotiation.cc:407] Connection header received
0929 11:26:16.438874 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.438878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.439047 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.439170 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.439600 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.440315 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.441871 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.442446 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.442588 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.443114 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.443643 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.443809 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.443859 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.443942 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:26:16.458146 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.449684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.0.97.62:37251)
0929 11:26:16.450404 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.450413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.451080 (+ 667us) server_negotiation.cc:407] Connection header received
0929 11:26:16.451484 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.451494 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.451747 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.451938 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.452723 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.453584 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.455634 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.456281 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.456400 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.457002 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.457473 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.457582 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.457726 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.457828 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":465,"threads_started":1}
I20260929 11:26:16.458726 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.449981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.0.97.61:41391)
0929 11:26:16.450189 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.450197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.451853 (+ 1656us) server_negotiation.cc:407] Connection header received
0929 11:26:16.451987 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.451994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.452091 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.452203 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.453091 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.454139 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.455952 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.456539 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.456712 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.457353 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.457473 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.457569 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.458236 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.458403 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:26:16.459172 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:16.449812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.0.97.60:34719)
0929 11:26:16.449991 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:16.449998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:16.451358 (+ 1360us) server_negotiation.cc:407] Connection header received
0929 11:26:16.451661 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:16.451668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:16.451757 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:16.451823 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:16.452945 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.454051 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.456115 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:16.456837 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:16.456958 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:16.457417 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:16.457562 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:16.457659 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:16.458792 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0929 11:26:16.458880 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:26:16.462668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 23175
I20260929 11:26:16.464037 23237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:16.574584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 23175
I20260929 11:26:16.596244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 23249
I20260929 11:26:16.597471 23311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:16.705333 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 23249
I20260929 11:26:16.725734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu pid 23323
I20260929 11:26:16.726802 23385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:16.835563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyNUK1V/build/asan/bin/kudu with pid 23323
2026-09-29T11:26:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4348 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9211 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (354938 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST